Sep 15 11:36:41.771914 ip-10-0-143-211 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 11:36:41.771925 ip-10-0-143-211 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 11:36:41.771934 ip-10-0-143-211 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 11:36:41.772253 ip-10-0-143-211 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 11:36:51.800676 ip-10-0-143-211 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 11:36:51.800694 ip-10-0-143-211 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 67c4a4172ac04a8e8c639d4c0ca425d5 --
Sep 15 11:38:31.097121 ip-10-0-143-211 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 11:38:31.510145 ip-10-0-143-211 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 11:38:31.510145 ip-10-0-143-211 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 11:38:31.510145 ip-10-0-143-211 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 11:38:31.510145 ip-10-0-143-211 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 11:38:31.510145 ip-10-0-143-211 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 11:38:31.512150 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.512065    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 11:38:31.515076 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515061    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515077    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515081    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515085    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515088    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515091    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515095    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515097    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515101    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515103    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515106    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515109    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515112    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515114    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515117    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:38:31.515112 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515120    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515123    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515126    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515128    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515131    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515140    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515143    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515146    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515149    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515151    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515154    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515157    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515159    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515162    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515164    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515167    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515172    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515175    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515179    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:38:31.515473 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515182    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515185    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515187    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515190    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515193    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515196    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515198    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515201    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515204    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515207    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515209    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515211    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515214    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515216    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515219    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515223    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515225    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515228    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515230    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515233    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:38:31.515962 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515235    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515238    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515240    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515244    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515246    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515249    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515254    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515258    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515261    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515263    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515266    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515269    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515271    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515274    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515277    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515279    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515282    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515285    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515287    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515290    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:38:31.516447 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515292    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515295    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515297    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515300    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515302    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515305    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515308    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515312    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515314    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515317    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515319    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515322    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515324    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515327    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515329    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515332    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515334    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515337    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515339    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515342    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:38:31.516946 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515344    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515347    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515349    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515351    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.515354    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516621    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516631    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516635    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516638    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516641    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516644    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516647    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516650    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516652    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516655    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516657    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516660    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516662    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516665    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516668    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:38:31.517470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516671    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516674    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516676    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516679    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516681    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516685    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516687    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516690    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516692    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516695    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516698    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516701    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516705    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516709    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516712    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516715    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516717    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516720    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516723    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516725    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:38:31.517978 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516728    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516730    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516733    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516736    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516738    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516741    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516743    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516746    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516748    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516751    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516754    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516757    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516760    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516762    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516765    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516767    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516770    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516773    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516775    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:38:31.518470 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516778    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516780    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516782    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516785    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516787    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516790    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516792    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516795    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516797    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516800    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516803    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516806    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516810    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516813    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516817    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516820    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516822    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516825    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516829    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:38:31.518958 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516832    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516834    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516836    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516839    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516842    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516845    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516847    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516850    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516853    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516855    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516858    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516860    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516863    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516865    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516883    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516886    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516889    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516892    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516894    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:38:31.519422 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516897    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516899    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516902    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516905    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516907    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516909    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.516912    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517265    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517270    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517273    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517276    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517281    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517285    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517288    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517291    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517294    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517297    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517299    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517302    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:38:31.519919 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517305    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517308    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517310    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517313    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517315    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517318    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517320    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517323    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517326    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517328    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517331    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517333    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517336    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517339    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517342    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517345    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517348    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517351    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517354    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517357    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:38:31.520376 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517360    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517363    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517366    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517369    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517372    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517375    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517378    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517381    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517383    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517386    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517389    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517392    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517394    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517397    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517399    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517402    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517404    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517407    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517409    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517412    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:38:31.520956 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517415    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517419    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517421    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517424    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517426    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517429    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517432    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517436    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517438    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517441    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517443    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517446    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517448    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517451    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517453    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517455    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517458    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517460    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517468    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517470    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:38:31.521607 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517473    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517476    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517479    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517482    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517485    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517487    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517490    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517492    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517495    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517497    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517500    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517502    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517505    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517507    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517510    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517512    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517514    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517517    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517519    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517522    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:38:31.522214 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517525    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517527    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517530    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517533    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517535    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517538    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.517540    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517631    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517679    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517686    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517691    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517696    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517700    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517705    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517709    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517712    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517715    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517719    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517722    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517726    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517729    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517732    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517735    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 15 11:38:31.522773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517738    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517741    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517746    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517749    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517752    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517756    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517759    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517766    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517769    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517773    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517776    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517779    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517782    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517786    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517789    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517792    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517796    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517799    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517802    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517805    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517808    2580 flags.go:64] FLAG: --enable-server="true"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517814    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517819    2580 flags.go:64] FLAG: --event-burst="100"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517822    2580 flags.go:64] FLAG: --event-qps="50"
Sep 15 11:38:31.523403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517825    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517828    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517831    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517834    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517837    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517840    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517843    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517846    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517849    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517852    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517855    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517858    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517861    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517864    2580 flags.go:64] FLAG: --feature-gates=""
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517879    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517885    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517888    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517892    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517895    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517898    2580 flags.go:64] FLAG: --help="false"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517901    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517904    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517907    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 11:38:31.524084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517911    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517914    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517917    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517920    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517923    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517926    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517929    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517932    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517936    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517939    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517942    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517944    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517947    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517950    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517953    2580 flags.go:64] FLAG: --lock-file=""
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517956    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517959    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517962    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517967    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517970    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517973    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517976    2580 flags.go:64] FLAG: --logging-format="text"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517979    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517983    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 11:38:31.524739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517987    2580 flags.go:64] FLAG: --manifest-url=""
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517992    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.517997    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518000    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518004    2580 flags.go:64] FLAG: --max-pods="110"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518007    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518010    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518013    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518016    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518020    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518022    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518025    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518032    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518036    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518039    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518042    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518046    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518051    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518054    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518057    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518060    2580 flags.go:64] FLAG: --port="10250"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518063    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518066    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f01da317523afd28"
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518070    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 15 11:38:31.525369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518073    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518076    2580 flags.go:64] FLAG: --register-node="true"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518078    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518082    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518085    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518088    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518091    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518094    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518098    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518102    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518106    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518109    2580 flags.go:64] FLAG: --runonce="false"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518112    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518115    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518118    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518121    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518124    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518127    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518130    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518133    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518136    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518139    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518142    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518145    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518148    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 11:38:31.525996 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518152    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518154    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518160    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518163    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518165    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518170    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518173    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518176    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518178    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518181    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518184    2580 flags.go:64] FLAG: --v="2"
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518188    2580 flags.go:64] FLAG: --version="false"
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518192    2580 flags.go:64] FLAG: --vmodule=""
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518195    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.518199    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518307    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518311    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518316    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518319    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518322    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518325    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518328    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518331    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:38:31.526627 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518333    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518336    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518339    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518342    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518344    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518347    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518349    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518354    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518357    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518361    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518364    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518367    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518371    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518374    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518377    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518380    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518383    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518385    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518388    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518391    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:38:31.527212 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518393    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518396    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518399    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518401    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518405    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518408    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518412    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518414    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518417    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518420    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518423    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518425    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518428    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518431    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518434    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518436    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518439    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518442    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518444    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518447    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:38:31.527716 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518450    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518452    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518455    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518458    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518461    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518463    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518466    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518469    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518471    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518474    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518477    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518480    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518482    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518485    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518488    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518490    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518494    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518496    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518500    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518503    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:38:31.528244 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518505    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518508    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518510    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518513    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518515    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518518    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518521    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518523    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518526    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518528    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518531    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518535    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518538    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518541    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518544    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518546    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518549    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518553    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518557    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:38:31.528733 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518560    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518563    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518566    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518569    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518571    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518574    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518577    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518579    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518582    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518585    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518588    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.518592    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.519287    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.525996    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.526010    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 11:38:31.529231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.526843    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.526850    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526917    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526922    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526925    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526929    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526932    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526935    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526938    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526941    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526944    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526947    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526950    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526952    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526956    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526959    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526962    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526965    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526967    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:38:31.529643 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526971    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526973    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526976    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526979    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526982    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526985    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526988    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526991    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526993    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526996    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.526998    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527001    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527005    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527008    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527011    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527013    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527016    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527019    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527021    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527024    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:38:31.530120 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527027    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527029    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527032    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527035    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527037    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527040    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527042    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527045    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527048    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527051    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527053    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527056    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527058    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527061    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527064    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527066    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527069    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527073    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527077    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527079    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:38:31.530650 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527082    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527085    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527087    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527090    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527094    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527098    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527102    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527105    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527108    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527111    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527114    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527117    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527120    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527122    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527125    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527128    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527130    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527134    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527138    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:38:31.531378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527141    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527144    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527147    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527150    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527153    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527156    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527160    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527163    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527166    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527169    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527171    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527174    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527177    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527180    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527183    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527186    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527189    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527192    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527194    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:38:31.531997 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527197    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527199    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527202    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527205    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.527210    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527334    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527339    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527342    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527345    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527348    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527351    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527353    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527356    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527359    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527361    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:38:31.532465 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527364    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527367    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527369    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527372    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527374    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527377    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527380    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527382    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527385    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527388    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527390    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527393    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527395    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527398    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527402    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527404    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527407    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527409    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527412    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527414    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:38:31.532853 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527417    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527419    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527422    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527426    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527429    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527432    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527435    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527438    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527441    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527444    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527447    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527451    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527454    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527457    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527460    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527463    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527466    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527468    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527471    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:38:31.533378 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527474    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527476    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527479    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527481    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527484    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527486    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527489    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527492    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527494    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527497    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527500    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527502    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527505    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527507    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527510    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527512    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527515    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527517    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527520    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527522    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:38:31.533932 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527525    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527528    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527530    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527533    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527535    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527538    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527540    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527543    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527549    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527551    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527554    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527557    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527559    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527562    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527564    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527567    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527569    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527572    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527575    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527577    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:38:31.534428 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527587    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:38:31.534955 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527592    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:38:31.534955 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527595    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:38:31.534955 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527597    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:38:31.534955 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527600    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:38:31.534955 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527602    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:38:31.534955 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527605    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:38:31.534955 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527607    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:38:31.534955 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527610    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:38:31.534955 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:31.527612    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:38:31.534955 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.527617    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 11:38:31.534955 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.527706    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 11:38:31.534955 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.531426    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 11:38:31.534955 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.532762    2580 server.go:1019] "Starting client certificate rotation"
Sep 15 11:38:31.534955 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.532885    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 11:38:31.535387 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.533626    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 11:38:31.535433 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.535415    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 11:38:31.536589 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.536569    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 11:38:31.546172 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.546154    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 15 11:38:31.549175 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.549159    2580 log.go:25] "Validated CRI v1 image API"
Sep 15 11:38:31.550091 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.550075    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 11:38:31.556075 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.556057    2580 fs.go:135] Filesystem UUIDs: map[693a204d-0491-44ba-87b3-660b0ba37ad0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f69a9551-4109-435b-a346-7bd085d71ae1:/dev/nvme0n1p4]
Sep 15 11:38:31.556139 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.556075    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 11:38:31.561481 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.561378    2580 manager.go:216] Machine: {Timestamp:2026-09-15 11:38:31.560007907 +0000 UTC m=+0.340579286 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098730 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26a9cfb22debe7a3cbea071c2632fe SystemUUID:ec26a9cf-b22d-ebe7-a3cb-ea071c2632fe BootID:67c4a417-2ac0-4a8e-8c63-9d4c0ca425d5 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ce:bf:0a:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ce:bf:0a:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:f7:af:d7:68:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 11:38:31.561481 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.561479    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 11:38:31.561620 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.561547    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 11:38:31.565232 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.565218    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 11:38:31.565292 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.565246    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 11:38:31.565404 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.565273    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-211.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 11:38:31.566059 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.566048    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 11:38:31.566091 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.566061    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 11:38:31.566091 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.566069    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 11:38:31.566597 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.566587    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 11:38:31.567468 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.567458    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 11:38:31.571326 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.571315    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 11:38:31.571435 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.571426    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 11:38:31.571464 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.571453    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 11:38:31.571464 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.571462    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 11:38:31.571517 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.571479    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 11:38:31.571517 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.571489    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 15 11:38:31.571517 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.571498    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 11:38:31.571953 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.571836    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 11:38:31.571993 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.571963    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 11:38:31.576652 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.576633    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 11:38:31.578961 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.578946    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 11:38:31.579023 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.578976    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 11:38:31.580561 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.580549    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 11:38:31.580615 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.580566    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 11:38:31.580615 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.580572    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 11:38:31.580615 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.580577    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 11:38:31.580615 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.580586    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 11:38:31.580615 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.580595    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 11:38:31.580615 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.580612    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 11:38:31.580615 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.580617    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 11:38:31.580823 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.580625    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 11:38:31.580823 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.580630    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 11:38:31.580823 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.580638    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 11:38:31.580823 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.580643    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 11:38:31.582342 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.582332    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 11:38:31.582379 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.582343    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 11:38:31.583211 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.583186    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 11:38:31.583281 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.583240    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-211.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 11:38:31.585009 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.584993    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-211.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 11:38:31.586160 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.586147    2580 server.go:1268] "Started kubelet"
Sep 15 11:38:31.586254 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.586228    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 11:38:31.586797 ip-10-0-143-211 systemd[1]: Started Kubernetes Kubelet.
Sep 15 11:38:31.587797 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.586800    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 11:38:31.587797 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.586858    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 11:38:31.588496 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.588479    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 11:38:31.589649 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.588919    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 11:38:31.592709 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.592688    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 11:38:31.593175 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.593156    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g4cx2"
Sep 15 11:38:31.595231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.595204    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 11:38:31.595737 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.595719    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 11:38:31.595983 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.595943    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-211.ec2.internal\" not found"
Sep 15 11:38:31.596320 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.596305    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 11:38:31.596407 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.596320    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 11:38:31.596407 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.596345    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 11:38:31.596509 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.596450    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 11:38:31.596509 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.596460    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 11:38:31.596509 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.596497    2580 factory.go:55] Registering systemd factory
Sep 15 11:38:31.596509 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.596513    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 15 11:38:31.596761 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.596750    2580 factory.go:153] Registering CRI-O factory
Sep 15 11:38:31.596798 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.596764    2580 factory.go:223] Registration of the crio container factory successfully
Sep 15 11:38:31.596839 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.596816    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 11:38:31.596839 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.596831    2580 factory.go:103] Registering Raw factory
Sep 15 11:38:31.596956 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.596849    2580 manager.go:1196] Started watching for new ooms in manager
Sep 15 11:38:31.597189 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.597176    2580 manager.go:349] Starting recovery of all containers
Sep 15 11:38:31.601091 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.601065    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 11:38:31.601181 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.601132    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-211.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 11:38:31.602476 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.601456    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-211.ec2.internal.18d57b961b7cd339  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-211.ec2.internal,UID:ip-10-0-143-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-211.ec2.internal,},FirstTimestamp:2026-09-15 11:38:31.586116409 +0000 UTC m=+0.366687788,LastTimestamp:2026-09-15 11:38:31.586116409 +0000 UTC m=+0.366687788,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-211.ec2.internal,}"
Sep 15 11:38:31.602586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.602553    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g4cx2"
Sep 15 11:38:31.609392 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.609370    2580 manager.go:354] Recovery completed
Sep 15 11:38:31.615540 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.615522    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:38:31.617402 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.617385    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:38:31.617487 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.617418    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:38:31.617487 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.617432    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:38:31.618060 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.618041    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 11:38:31.618158 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.618060    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 11:38:31.618158 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.618078    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 11:38:31.618158 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.618124    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:38:31.618991 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.618973    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:38:31.619062 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.619003    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:38:31.619062 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.619017    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:38:31.619516 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.619454    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-211.ec2.internal.18d57b961d5a31d8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-211.ec2.internal,UID:ip-10-0-143-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-211.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-211.ec2.internal,},FirstTimestamp:2026-09-15 11:38:31.617401304 +0000 UTC m=+0.397972687,LastTimestamp:2026-09-15 11:38:31.617401304 +0000 UTC m=+0.397972687,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-211.ec2.internal,}"
Sep 15 11:38:31.621255 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.621241    2580 policy_none.go:49] "None policy: Start"
Sep 15 11:38:31.621305 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.621258    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 11:38:31.621305 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.621268    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 11:38:31.623299 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.623288    2580 policy_none.go:47] "Start"
Sep 15 11:38:31.651432 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.651416    2580 manager.go:337] "Starting Device Plugin manager"
Sep 15 11:38:31.672917 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.651461    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 11:38:31.672917 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.651475    2580 server.go:87] "Starting device plugin registration server"
Sep 15 11:38:31.672917 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.651768    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 11:38:31.672917 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.651778    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 11:38:31.672917 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.651866    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 11:38:31.672917 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.651964    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 11:38:31.672917 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.651972    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 11:38:31.672917 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.652518    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 11:38:31.672917 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.652578    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-211.ec2.internal\" not found"
Sep 15 11:38:31.722914 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.722887    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 11:38:31.724056 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.724042    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 11:38:31.724127 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.724086    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 11:38:31.724127 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.724108    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 11:38:31.724208 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.724150    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 11:38:31.726661 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.726643    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:38:31.752690 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.752672    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:38:31.753577 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.753557    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:38:31.753704 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.753592    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:38:31.753704 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.753606    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:38:31.753704 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.753634    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.762811 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.762734    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.762811 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.762755    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-211.ec2.internal\": node \"ip-10-0-143-211.ec2.internal\" not found"
Sep 15 11:38:31.779191 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.779168    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-211.ec2.internal\" not found"
Sep 15 11:38:31.824642 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.824623    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-211.ec2.internal"]
Sep 15 11:38:31.824736 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.824680    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:38:31.826624 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.826608    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:38:31.826716 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.826635    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:38:31.826716 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.826650    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:38:31.829144 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.829130    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:38:31.829266 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.829252    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.829322 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.829284    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:38:31.829918 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.829898    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:38:31.829998 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.829932    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:38:31.829998 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.829938    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:38:31.829998 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.829944    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:38:31.829998 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.829961    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:38:31.829998 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.829983    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:38:31.832231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.832217    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.832305 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.832239    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:38:31.832959 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.832943    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:38:31.833020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.832968    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:38:31.833020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.832980    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:38:31.845423 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.845405    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-211.ec2.internal\" not found" node="ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.849294 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.849281    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-211.ec2.internal\" not found" node="ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.880140 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.880121    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-211.ec2.internal\" not found"
Sep 15 11:38:31.898805 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.898730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/883a21e7077324fc24e91cbc981bfd76-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal\" (UID: \"883a21e7077324fc24e91cbc981bfd76\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.898884 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.898814    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/883a21e7077324fc24e91cbc981bfd76-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal\" (UID: \"883a21e7077324fc24e91cbc981bfd76\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.898884 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.898830    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/acc80c84a62ea6b15d405529c0cb63ed-config\") pod \"kube-apiserver-proxy-ip-10-0-143-211.ec2.internal\" (UID: \"acc80c84a62ea6b15d405529c0cb63ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.981153 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:31.981126    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-211.ec2.internal\" not found"
Sep 15 11:38:31.999524 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.999502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/883a21e7077324fc24e91cbc981bfd76-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal\" (UID: \"883a21e7077324fc24e91cbc981bfd76\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.999608 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.999528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/883a21e7077324fc24e91cbc981bfd76-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal\" (UID: \"883a21e7077324fc24e91cbc981bfd76\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.999608 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.999547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/acc80c84a62ea6b15d405529c0cb63ed-config\") pod \"kube-apiserver-proxy-ip-10-0-143-211.ec2.internal\" (UID: \"acc80c84a62ea6b15d405529c0cb63ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.999684 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.999609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/acc80c84a62ea6b15d405529c0cb63ed-config\") pod \"kube-apiserver-proxy-ip-10-0-143-211.ec2.internal\" (UID: \"acc80c84a62ea6b15d405529c0cb63ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.999684 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.999611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/883a21e7077324fc24e91cbc981bfd76-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal\" (UID: \"883a21e7077324fc24e91cbc981bfd76\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:31.999684 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:31.999617    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/883a21e7077324fc24e91cbc981bfd76-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal\" (UID: \"883a21e7077324fc24e91cbc981bfd76\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:32.081986 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:32.081930    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-211.ec2.internal\" not found"
Sep 15 11:38:32.148464 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:32.148433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:32.151914 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:32.151896    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:32.182619 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:32.182596    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-211.ec2.internal\" not found"
Sep 15 11:38:32.286566 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:32.286531    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-211.ec2.internal\" not found"
Sep 15 11:38:32.387418 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:32.387359    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-211.ec2.internal\" not found"
Sep 15 11:38:32.489601 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:32.489569    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-211.ec2.internal\" not found"
Sep 15 11:38:32.533385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:32.533356    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 11:38:32.533790 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:32.533509    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 11:38:32.589992 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:32.589968    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-211.ec2.internal\" not found"
Sep 15 11:38:32.607613 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:32.607574    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 11:33:32 +0000 UTC" deadline="2028-04-28 21:31:54.732810119 +0000 UTC"
Sep 15 11:38:32.607613 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:32.607602    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14193h53m22.12521116s"
Sep 15 11:38:32.663836 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:32.663770    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:38:32.691037 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:32.691009    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-211.ec2.internal\" not found"
Sep 15 11:38:32.785411 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:32.785371    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc80c84a62ea6b15d405529c0cb63ed.slice/crio-5b83060b58f61cf6fbbffa224b5998468bc9f9b05fab012643887336ab9d35a5 WatchSource:0}: Error finding container 5b83060b58f61cf6fbbffa224b5998468bc9f9b05fab012643887336ab9d35a5: Status 404 returned error can't find the container with id 5b83060b58f61cf6fbbffa224b5998468bc9f9b05fab012643887336ab9d35a5
Sep 15 11:38:32.785783 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:32.785757    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883a21e7077324fc24e91cbc981bfd76.slice/crio-75e2eb9edcc0404d326295eacedfb38889cca1e7b92a7a3c1dd41d06fcf282b9 WatchSource:0}: Error finding container 75e2eb9edcc0404d326295eacedfb38889cca1e7b92a7a3c1dd41d06fcf282b9: Status 404 returned error can't find the container with id 75e2eb9edcc0404d326295eacedfb38889cca1e7b92a7a3c1dd41d06fcf282b9
Sep 15 11:38:32.789931 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:32.789912    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 11:38:32.791782 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:32.791758    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-211.ec2.internal\" not found"
Sep 15 11:38:32.805257 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:32.805239    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:38:32.897837 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:32.897814    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:32.911160 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:32.911140    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 11:38:32.913073 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:32.913058    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-211.ec2.internal"
Sep 15 11:38:32.923760 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:32.923717    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 11:38:32.942432 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:32.942412    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:38:33.573404 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.573161    2580 apiserver.go:52] "Watching apiserver"
Sep 15 11:38:33.587726 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.587693    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 11:38:33.588255 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.588202    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal","openshift-multus/multus-additional-cni-plugins-b899l","openshift-multus/multus-vnm5x","openshift-network-diagnostics/network-check-target-dvh4t","kube-system/kube-apiserver-proxy-ip-10-0-143-211.ec2.internal","openshift-dns/node-resolver-wjqmg","openshift-image-registry/node-ca-hd8pm","openshift-multus/network-metrics-daemon-rxkvf","openshift-network-operator/iptables-alerter-hr2q4","openshift-ovn-kubernetes/ovnkube-node-4nqs5","kube-system/global-pull-secret-syncer-6tnl8","kube-system/konnectivity-agent-pdj4g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9","openshift-cluster-node-tuning-operator/tuned-ptgh5"]
Sep 15 11:38:33.591299 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.591271    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hd8pm"
Sep 15 11:38:33.594584 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.594562    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:33.594694 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:33.594641    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:38:33.595110 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.595089    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 11:38:33.595212 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.595108    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 11:38:33.595408 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.595388    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gjgj\""
Sep 15 11:38:33.595486 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.595412    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 11:38:33.598026 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.598003    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.599068 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.599046    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:33.599165 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:33.599111    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:38:33.599165 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.599162    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 11:38:33.600315 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.600259    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 11:38:33.600720 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.600703    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 11:38:33.600856 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.600811    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 11:38:33.601151 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.601116    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rsrml\""
Sep 15 11:38:33.601239 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.601152    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 11:38:33.601894 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.601463    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.603859 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.603833    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wjqmg"
Sep 15 11:38:33.606801 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.604481    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 11:38:33.606801 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.604813    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lvzb7\""
Sep 15 11:38:33.606801 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.605061    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:38:33.607029 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.606973    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lvncp\""
Sep 15 11:38:33.609390 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.607209    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 11:38:33.609390 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.607295    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 11:38:33.609390 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.607478    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hr2q4"
Sep 15 11:38:33.609390 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.608782    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.610003 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.609776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c213c82d-7bda-4c00-a867-059ab500ca55-cni-binary-copy\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.610003 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.609843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-etc-kubernetes\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.610003 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.609880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec648b53-d8f0-43aa-82d3-5ae9212975d6-kubelet-config\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:33.610003 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.609898    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:38:33.610003 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.609910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-run-netns\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.610003 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.609937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-run-multus-certs\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.610003 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.609994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mv54\" (UniqueName: \"kubernetes.io/projected/c213c82d-7bda-4c00-a867-059ab500ca55-kube-api-access-7mv54\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.610443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk99l\" (UniqueName: \"kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l\") pod \"network-check-target-dvh4t\" (UID: \"b14e7c6d-9418-49ca-81f8-2b81ed8bce7b\") " pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:33.610443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdjwl\" (UniqueName: \"kubernetes.io/projected/d00a7cd2-9093-4f86-b40b-1b29b8f6d061-kube-api-access-wdjwl\") pod \"node-ca-hd8pm\" (UID: \"d00a7cd2-9093-4f86-b40b-1b29b8f6d061\") " pod="openshift-image-registry/node-ca-hd8pm"
Sep 15 11:38:33.610443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610172    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-system-cni-dir\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.610443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-multus-socket-dir-parent\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.610443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-var-lib-cni-bin\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.610443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-var-lib-cni-multus\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.610443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-var-lib-kubelet\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.610443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-hostroot\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.610443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d00a7cd2-9093-4f86-b40b-1b29b8f6d061-serviceca\") pod \"node-ca-hd8pm\" (UID: \"d00a7cd2-9093-4f86-b40b-1b29b8f6d061\") " pod="openshift-image-registry/node-ca-hd8pm"
Sep 15 11:38:33.610443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec648b53-d8f0-43aa-82d3-5ae9212975d6-dbus\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:33.610443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-cnibin\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.610443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610385    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 11:38:33.610443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-run-k8s-cni-cncf-io\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.611163 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-multus-conf-dir\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.611163 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c213c82d-7bda-4c00-a867-059ab500ca55-multus-daemon-config\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.611163 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d00a7cd2-9093-4f86-b40b-1b29b8f6d061-host\") pod \"node-ca-hd8pm\" (UID: \"d00a7cd2-9093-4f86-b40b-1b29b8f6d061\") " pod="openshift-image-registry/node-ca-hd8pm"
Sep 15 11:38:33.611163 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:33.611163 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.610623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-multus-cni-dir\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.611421 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.611209    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 11:38:33.612112 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.612083    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 11:38:33.612328 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.612306    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 11:38:33.612550 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.612532    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fql47\""
Sep 15 11:38:33.612778 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.612760    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tlgxd\""
Sep 15 11:38:33.612857 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.612793    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.613821 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.613612    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 11:38:33.615464 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.615306    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qdc4d\""
Sep 15 11:38:33.617224 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.615788    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 11:38:33.617224 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.616058    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 11:38:33.617224 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.616722    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.617224 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.616722    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 11:38:33.619490 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.619472    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 11:38:33.619817 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.619800    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-844l7\""
Sep 15 11:38:33.620337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.620075    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:33.620337 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:33.620139    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:38:33.620928 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.620597    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 11:38:33.620928 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.620651    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 11:38:33.620928 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.620798    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 11:38:33.621231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.621215    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 11:38:33.621494 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.621475    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 11:38:33.622205 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.621863    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pdj4g"
Sep 15 11:38:33.624982 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.624144    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 11:38:33.624982 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.624420    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 11:38:33.624982 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.624603    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dfnd5\""
Sep 15 11:38:33.642471 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.642443    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2pf7j"
Sep 15 11:38:33.649032 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.649008    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2pf7j"
Sep 15 11:38:33.695387 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.695363    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:38:33.697557 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.697538    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 11:38:33.710832 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.710808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5467c994-9999-4b50-880d-538763991a9e-tmp-dir\") pod \"node-resolver-wjqmg\" (UID: \"5467c994-9999-4b50-880d-538763991a9e\") " pod="openshift-dns/node-resolver-wjqmg"
Sep 15 11:38:33.710961 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.710838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.710961 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.710860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.710961 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.710898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-ovn-node-metrics-cert\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.710961 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.710930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d00a7cd2-9093-4f86-b40b-1b29b8f6d061-serviceca\") pod \"node-ca-hd8pm\" (UID: \"d00a7cd2-9093-4f86-b40b-1b29b8f6d061\") " pod="openshift-image-registry/node-ca-hd8pm"
Sep 15 11:38:33.710961 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.710953    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-modprobe-d\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.711217 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-tmp\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.711217 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-device-dir\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.711217 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-etc-selinux\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.711217 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-sys-fs\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.711217 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.711217 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711174    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d00a7cd2-9093-4f86-b40b-1b29b8f6d061-host\") pod \"node-ca-hd8pm\" (UID: \"d00a7cd2-9093-4f86-b40b-1b29b8f6d061\") " pod="openshift-image-registry/node-ca-hd8pm"
Sep 15 11:38:33.711217 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:33.711217 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c213c82d-7bda-4c00-a867-059ab500ca55-cni-binary-copy\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-systemd-units\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-var-lib-openvswitch\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711265    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-log-socket\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-cni-bin\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711295    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-ovnkube-script-lib\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-systemd\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-sys\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-run-ovn\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-var-lib-cni-bin\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-hostroot\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d00a7cd2-9093-4f86-b40b-1b29b8f6d061-serviceca\") pod \"node-ca-hd8pm\" (UID: \"d00a7cd2-9093-4f86-b40b-1b29b8f6d061\") " pod="openshift-image-registry/node-ca-hd8pm"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs\") pod \"network-metrics-daemon-rxkvf\" (UID: \"1263d93a-e6b5-497d-a022-f24619717fa9\") " pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d00a7cd2-9093-4f86-b40b-1b29b8f6d061-host\") pod \"node-ca-hd8pm\" (UID: \"d00a7cd2-9093-4f86-b40b-1b29b8f6d061\") " pod="openshift-image-registry/node-ca-hd8pm"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-socket-dir\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-run-openvswitch\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.711621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-hostroot\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:33.711629    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-node-log\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:33.711719    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret podName:ec648b53-d8f0-43aa-82d3-5ae9212975d6 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:34.211681219 +0000 UTC m=+2.992252593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret") pod "global-pull-secret-syncer-6tnl8" (UID: "ec648b53-d8f0-43aa-82d3-5ae9212975d6") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-var-lib-cni-bin\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-multus-conf-dir\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-sysconfig\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/81aebea3-e5e1-4326-bfb2-1bec4e195109-iptables-alerter-script\") pod \"iptables-alerter-hr2q4\" (UID: \"81aebea3-e5e1-4326-bfb2-1bec4e195109\") " pod="openshift-network-operator/iptables-alerter-hr2q4"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-multus-conf-dir\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-etc-kubernetes\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x5dz\" (UniqueName: \"kubernetes.io/projected/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-kube-api-access-9x5dz\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.711991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-etc-openvswitch\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-run-multus-certs\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c213c82d-7bda-4c00-a867-059ab500ca55-cni-binary-copy\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ed568cb-25cc-452a-a618-ef7b524de14a-konnectivity-ca\") pod \"konnectivity-agent-pdj4g\" (UID: \"3ed568cb-25cc-452a-a618-ef7b524de14a\") " pod="kube-system/konnectivity-agent-pdj4g"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-slash\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712110    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-etc-kubernetes\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.712429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-run-netns\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712150    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-run-multus-certs\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdjwl\" (UniqueName: \"kubernetes.io/projected/d00a7cd2-9093-4f86-b40b-1b29b8f6d061-kube-api-access-wdjwl\") pod \"node-ca-hd8pm\" (UID: \"d00a7cd2-9093-4f86-b40b-1b29b8f6d061\") " pod="openshift-image-registry/node-ca-hd8pm"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-system-cni-dir\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-multus-socket-dir-parent\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-var-lib-cni-multus\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-var-lib-kubelet\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mv54\" (UniqueName: \"kubernetes.io/projected/c213c82d-7bda-4c00-a867-059ab500ca55-kube-api-access-7mv54\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5467c994-9999-4b50-880d-538763991a9e-hosts-file\") pod \"node-resolver-wjqmg\" (UID: \"5467c994-9999-4b50-880d-538763991a9e\") " pod="openshift-dns/node-resolver-wjqmg"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-system-cni-dir\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-multus-socket-dir-parent\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-var-lib-kubelet\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ed568cb-25cc-452a-a618-ef7b524de14a-agent-certs\") pod \"konnectivity-agent-pdj4g\" (UID: \"3ed568cb-25cc-452a-a618-ef7b524de14a\") " pod="kube-system/konnectivity-agent-pdj4g"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-system-cni-dir\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712621    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-kubelet\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-run-systemd\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-var-lib-cni-multus\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.713337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-ovnkube-config\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-host\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4tks\" (UniqueName: \"kubernetes.io/projected/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-kube-api-access-b4tks\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-kubernetes\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-sysctl-d\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.712985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-lib-modules\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-tuned\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81aebea3-e5e1-4326-bfb2-1bec4e195109-host-slash\") pod \"iptables-alerter-hr2q4\" (UID: \"81aebea3-e5e1-4326-bfb2-1bec4e195109\") " pod="openshift-network-operator/iptables-alerter-hr2q4"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfwqg\" (UniqueName: \"kubernetes.io/projected/81aebea3-e5e1-4326-bfb2-1bec4e195109-kube-api-access-zfwqg\") pod \"iptables-alerter-hr2q4\" (UID: \"81aebea3-e5e1-4326-bfb2-1bec4e195109\") " pod="openshift-network-operator/iptables-alerter-hr2q4"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-env-overrides\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c213c82d-7bda-4c00-a867-059ab500ca55-multus-daemon-config\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec648b53-d8f0-43aa-82d3-5ae9212975d6-dbus\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-cnibin\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-run-k8s-cni-cncf-io\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tvqf\" (UniqueName: \"kubernetes.io/projected/27a407e8-ad7e-4047-aff1-31aa68ac6021-kube-api-access-4tvqf\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.714190 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-cnibin\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-multus-cni-dir\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-var-lib-kubelet\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec648b53-d8f0-43aa-82d3-5ae9212975d6-kubelet-config\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-run-netns\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zgd5\" (UniqueName: \"kubernetes.io/projected/5467c994-9999-4b50-880d-538763991a9e-kube-api-access-2zgd5\") pod \"node-resolver-wjqmg\" (UID: \"5467c994-9999-4b50-880d-538763991a9e\") " pod="openshift-dns/node-resolver-wjqmg"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt8tl\" (UniqueName: \"kubernetes.io/projected/1263d93a-e6b5-497d-a022-f24619717fa9-kube-api-access-bt8tl\") pod \"network-metrics-daemon-rxkvf\" (UID: \"1263d93a-e6b5-497d-a022-f24619717fa9\") " pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-sysctl-conf\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-run\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wwg\" (UniqueName: \"kubernetes.io/projected/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-kube-api-access-t8wwg\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-cni-netd\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-registration-dir\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk99l\" (UniqueName: \"kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l\") pod \"network-check-target-dvh4t\" (UID: \"b14e7c6d-9418-49ca-81f8-2b81ed8bce7b\") " pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:33.715034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec648b53-d8f0-43aa-82d3-5ae9212975d6-dbus\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:33.715916 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.713984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-cnibin\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.715916 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.714019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-run-k8s-cni-cncf-io\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.715916 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.714091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-multus-cni-dir\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.715916 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.714146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec648b53-d8f0-43aa-82d3-5ae9212975d6-kubelet-config\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:33.715916 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.714175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c213c82d-7bda-4c00-a867-059ab500ca55-host-run-netns\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.715916 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.714363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c213c82d-7bda-4c00-a867-059ab500ca55-multus-daemon-config\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.720295 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:33.720273    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:38:33.720415 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:33.720306    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:38:33.720415 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:33.720320    2580 projected.go:196] Error preparing data for projected volume kube-api-access-tk99l for pod openshift-network-diagnostics/network-check-target-dvh4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:38:33.720415 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:33.720398    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l podName:b14e7c6d-9418-49ca-81f8-2b81ed8bce7b nodeName:}" failed. No retries permitted until 2026-09-15 11:38:34.220379076 +0000 UTC m=+3.000950450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tk99l" (UniqueName: "kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l") pod "network-check-target-dvh4t" (UID: "b14e7c6d-9418-49ca-81f8-2b81ed8bce7b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:38:33.720801 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.720776    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 11:38:33.726681 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.725976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdjwl\" (UniqueName: \"kubernetes.io/projected/d00a7cd2-9093-4f86-b40b-1b29b8f6d061-kube-api-access-wdjwl\") pod \"node-ca-hd8pm\" (UID: \"d00a7cd2-9093-4f86-b40b-1b29b8f6d061\") " pod="openshift-image-registry/node-ca-hd8pm"
Sep 15 11:38:33.726681 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.725983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mv54\" (UniqueName: \"kubernetes.io/projected/c213c82d-7bda-4c00-a867-059ab500ca55-kube-api-access-7mv54\") pod \"multus-vnm5x\" (UID: \"c213c82d-7bda-4c00-a867-059ab500ca55\") " pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.732339 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.732295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-211.ec2.internal" event={"ID":"acc80c84a62ea6b15d405529c0cb63ed","Type":"ContainerStarted","Data":"5b83060b58f61cf6fbbffa224b5998468bc9f9b05fab012643887336ab9d35a5"}
Sep 15 11:38:33.733869 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.733842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal" event={"ID":"883a21e7077324fc24e91cbc981bfd76","Type":"ContainerStarted","Data":"75e2eb9edcc0404d326295eacedfb38889cca1e7b92a7a3c1dd41d06fcf282b9"}
Sep 15 11:38:33.814454 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-systemd-units\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.814454 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-var-lib-openvswitch\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.814454 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-log-socket\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.814743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-cni-bin\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.814743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-systemd-units\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.814743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-ovnkube-script-lib\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.814743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-systemd\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.814743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-sys\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.814743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-run-ovn\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.814743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs\") pod \"network-metrics-daemon-rxkvf\" (UID: \"1263d93a-e6b5-497d-a022-f24619717fa9\") " pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:33.814743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-log-socket\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.814743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-socket-dir\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.814743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-var-lib-openvswitch\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.814743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-run-openvswitch\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-node-log\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-sys\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-run-openvswitch\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-sysconfig\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/81aebea3-e5e1-4326-bfb2-1bec4e195109-iptables-alerter-script\") pod \"iptables-alerter-hr2q4\" (UID: \"81aebea3-e5e1-4326-bfb2-1bec4e195109\") " pod="openshift-network-operator/iptables-alerter-hr2q4"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-cni-bin\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-systemd\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.814988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x5dz\" (UniqueName: \"kubernetes.io/projected/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-kube-api-access-9x5dz\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-etc-openvswitch\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ed568cb-25cc-452a-a618-ef7b524de14a-konnectivity-ca\") pod \"konnectivity-agent-pdj4g\" (UID: \"3ed568cb-25cc-452a-a618-ef7b524de14a\") " pod="kube-system/konnectivity-agent-pdj4g"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-socket-dir\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-slash\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:33.814991    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-run-netns\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:33.815154    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs podName:1263d93a-e6b5-497d-a022-f24619717fa9 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:34.315128344 +0000 UTC m=+3.095699727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs") pod "network-metrics-daemon-rxkvf" (UID: "1263d93a-e6b5-497d-a022-f24619717fa9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5467c994-9999-4b50-880d-538763991a9e-hosts-file\") pod \"node-resolver-wjqmg\" (UID: \"5467c994-9999-4b50-880d-538763991a9e\") " pod="openshift-dns/node-resolver-wjqmg"
Sep 15 11:38:33.815385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-run-netns\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ed568cb-25cc-452a-a618-ef7b524de14a-agent-certs\") pod \"konnectivity-agent-pdj4g\" (UID: \"3ed568cb-25cc-452a-a618-ef7b524de14a\") " pod="kube-system/konnectivity-agent-pdj4g"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-etc-openvswitch\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-system-cni-dir\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-kubelet\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-run-systemd\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-ovnkube-config\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-host\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815527    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4tks\" (UniqueName: \"kubernetes.io/projected/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-kube-api-access-b4tks\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-kubernetes\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-sysctl-d\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-lib-modules\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-tuned\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-ovnkube-script-lib\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81aebea3-e5e1-4326-bfb2-1bec4e195109-host-slash\") pod \"iptables-alerter-hr2q4\" (UID: \"81aebea3-e5e1-4326-bfb2-1bec4e195109\") " pod="openshift-network-operator/iptables-alerter-hr2q4"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/81aebea3-e5e1-4326-bfb2-1bec4e195109-iptables-alerter-script\") pod \"iptables-alerter-hr2q4\" (UID: \"81aebea3-e5e1-4326-bfb2-1bec4e195109\") " pod="openshift-network-operator/iptables-alerter-hr2q4"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfwqg\" (UniqueName: \"kubernetes.io/projected/81aebea3-e5e1-4326-bfb2-1bec4e195109-kube-api-access-zfwqg\") pod \"iptables-alerter-hr2q4\" (UID: \"81aebea3-e5e1-4326-bfb2-1bec4e195109\") " pod="openshift-network-operator/iptables-alerter-hr2q4"
Sep 15 11:38:33.816290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-sysconfig\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-env-overrides\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-node-log\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815836    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-slash\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tvqf\" (UniqueName: \"kubernetes.io/projected/27a407e8-ad7e-4047-aff1-31aa68ac6021-kube-api-access-4tvqf\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-cnibin\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.815990    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-var-lib-kubelet\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ed568cb-25cc-452a-a618-ef7b524de14a-konnectivity-ca\") pod \"konnectivity-agent-pdj4g\" (UID: \"3ed568cb-25cc-452a-a618-ef7b524de14a\") " pod="kube-system/konnectivity-agent-pdj4g"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zgd5\" (UniqueName: \"kubernetes.io/projected/5467c994-9999-4b50-880d-538763991a9e-kube-api-access-2zgd5\") pod \"node-resolver-wjqmg\" (UID: \"5467c994-9999-4b50-880d-538763991a9e\") " pod="openshift-dns/node-resolver-wjqmg"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-host\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt8tl\" (UniqueName: \"kubernetes.io/projected/1263d93a-e6b5-497d-a022-f24619717fa9-kube-api-access-bt8tl\") pod \"network-metrics-daemon-rxkvf\" (UID: \"1263d93a-e6b5-497d-a022-f24619717fa9\") " pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-sysctl-conf\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-run\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.817279 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-run-ovn\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.818171 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wwg\" (UniqueName: \"kubernetes.io/projected/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-kube-api-access-t8wwg\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.818171 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-cni-netd\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.818171 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5467c994-9999-4b50-880d-538763991a9e-hosts-file\") pod \"node-resolver-wjqmg\" (UID: \"5467c994-9999-4b50-880d-538763991a9e\") " pod="openshift-dns/node-resolver-wjqmg"
Sep 15 11:38:33.818171 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-registration-dir\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.818171 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-kubernetes\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.818171 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5467c994-9999-4b50-880d-538763991a9e-tmp-dir\") pod \"node-resolver-wjqmg\" (UID: \"5467c994-9999-4b50-880d-538763991a9e\") " pod="openshift-dns/node-resolver-wjqmg"
Sep 15 11:38:33.818171 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.818171 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.818171 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-ovn-node-metrics-cert\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.818171 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816629    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-sysctl-d\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.818171 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.817328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-modprobe-d\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.818921 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.818888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-cnibin\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.819056 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.818978    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-cni-netd\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.819130 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.819050    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-var-lib-kubelet\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.819130 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.819057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-kubelet\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.819638 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.819611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.819761 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.819743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-lib-modules\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.821862 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.820142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-system-cni-dir\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.821862 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.816574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-run-systemd\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.821862 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.821095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-env-overrides\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.821862 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.821186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.821862 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.821811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-ovnkube-config\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.821862 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.821827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-tuned\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.822559 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.822300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-run\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.822559 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.822366    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-sysctl-conf\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.822559 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.822454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.822793 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.822628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.822793 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.822692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81aebea3-e5e1-4326-bfb2-1bec4e195109-host-slash\") pod \"iptables-alerter-hr2q4\" (UID: \"81aebea3-e5e1-4326-bfb2-1bec4e195109\") " pod="openshift-network-operator/iptables-alerter-hr2q4"
Sep 15 11:38:33.822793 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.822713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ed568cb-25cc-452a-a618-ef7b524de14a-agent-certs\") pod \"konnectivity-agent-pdj4g\" (UID: \"3ed568cb-25cc-452a-a618-ef7b524de14a\") " pod="kube-system/konnectivity-agent-pdj4g"
Sep 15 11:38:33.822793 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.822736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-tmp\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.823020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.822813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-device-dir\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.823020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.822854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-etc-selinux\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.823020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.822893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-registration-dir\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.823020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.822889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-sys-fs\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.823020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.822953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.823020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.822966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-sys-fs\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.823331 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.823153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5467c994-9999-4b50-880d-538763991a9e-tmp-dir\") pod \"node-resolver-wjqmg\" (UID: \"5467c994-9999-4b50-880d-538763991a9e\") " pod="openshift-dns/node-resolver-wjqmg"
Sep 15 11:38:33.823331 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.823244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.825053 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.823534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.825053 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.823613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-device-dir\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.825053 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.823714    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.825053 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.823839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27a407e8-ad7e-4047-aff1-31aa68ac6021-etc-selinux\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.825053 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.824053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-etc-modprobe-d\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.832637 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.832598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-tmp\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.832811 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.832789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-ovn-node-metrics-cert\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.836660 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.836633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tvqf\" (UniqueName: \"kubernetes.io/projected/27a407e8-ad7e-4047-aff1-31aa68ac6021-kube-api-access-4tvqf\") pod \"aws-ebs-csi-driver-node-wqcj9\" (UID: \"27a407e8-ad7e-4047-aff1-31aa68ac6021\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.836784 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.836760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x5dz\" (UniqueName: \"kubernetes.io/projected/0712213d-ab9e-4d03-ba6f-cbc70b17dcdb-kube-api-access-9x5dz\") pod \"tuned-ptgh5\" (UID: \"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb\") " pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.838260 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.838238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfwqg\" (UniqueName: \"kubernetes.io/projected/81aebea3-e5e1-4326-bfb2-1bec4e195109-kube-api-access-zfwqg\") pod \"iptables-alerter-hr2q4\" (UID: \"81aebea3-e5e1-4326-bfb2-1bec4e195109\") " pod="openshift-network-operator/iptables-alerter-hr2q4"
Sep 15 11:38:33.840651 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.840482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4tks\" (UniqueName: \"kubernetes.io/projected/665a9fc2-c525-47c2-9a83-9e1bd3af8ab0-kube-api-access-b4tks\") pod \"ovnkube-node-4nqs5\" (UID: \"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.840843 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.840823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zgd5\" (UniqueName: \"kubernetes.io/projected/5467c994-9999-4b50-880d-538763991a9e-kube-api-access-2zgd5\") pod \"node-resolver-wjqmg\" (UID: \"5467c994-9999-4b50-880d-538763991a9e\") " pod="openshift-dns/node-resolver-wjqmg"
Sep 15 11:38:33.841121 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.841100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wwg\" (UniqueName: \"kubernetes.io/projected/4cc598d7-b2ab-457d-a6e1-9ae73188f8e5-kube-api-access-t8wwg\") pod \"multus-additional-cni-plugins-b899l\" (UID: \"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5\") " pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.842486 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.842443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt8tl\" (UniqueName: \"kubernetes.io/projected/1263d93a-e6b5-497d-a022-f24619717fa9-kube-api-access-bt8tl\") pod \"network-metrics-daemon-rxkvf\" (UID: \"1263d93a-e6b5-497d-a022-f24619717fa9\") " pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:33.910524 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.910483    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hd8pm"
Sep 15 11:38:33.920049 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.920020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vnm5x"
Sep 15 11:38:33.933287 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.933262    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ptgh5"
Sep 15 11:38:33.940174 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.940147    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wjqmg"
Sep 15 11:38:33.950051 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.950032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hr2q4"
Sep 15 11:38:33.959028 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.959006    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9"
Sep 15 11:38:33.967909 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.967890    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b899l"
Sep 15 11:38:33.976881 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.976726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:33.983635 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:33.983615    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pdj4g"
Sep 15 11:38:34.226678 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:34.226596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk99l\" (UniqueName: \"kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l\") pod \"network-check-target-dvh4t\" (UID: \"b14e7c6d-9418-49ca-81f8-2b81ed8bce7b\") " pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:34.226678 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:34.226646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:34.226917 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:34.226770    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:38:34.226917 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:34.226794    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:38:34.226917 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:34.226794    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:38:34.226917 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:34.226805    2580 projected.go:196] Error preparing data for projected volume kube-api-access-tk99l for pod openshift-network-diagnostics/network-check-target-dvh4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:38:34.226917 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:34.226899    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret podName:ec648b53-d8f0-43aa-82d3-5ae9212975d6 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:35.226849244 +0000 UTC m=+4.007420612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret") pod "global-pull-secret-syncer-6tnl8" (UID: "ec648b53-d8f0-43aa-82d3-5ae9212975d6") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:38:34.227123 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:34.226924    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l podName:b14e7c6d-9418-49ca-81f8-2b81ed8bce7b nodeName:}" failed. No retries permitted until 2026-09-15 11:38:35.226913355 +0000 UTC m=+4.007484722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk99l" (UniqueName: "kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l") pod "network-check-target-dvh4t" (UID: "b14e7c6d-9418-49ca-81f8-2b81ed8bce7b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:38:34.327724 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:34.327686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs\") pod \"network-metrics-daemon-rxkvf\" (UID: \"1263d93a-e6b5-497d-a022-f24619717fa9\") " pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:34.327925 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:34.327830    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:38:34.327925 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:34.327896    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs podName:1263d93a-e6b5-497d-a022-f24619717fa9 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:35.327874222 +0000 UTC m=+4.108445605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs") pod "network-metrics-daemon-rxkvf" (UID: "1263d93a-e6b5-497d-a022-f24619717fa9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:38:34.653481 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:34.653376    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 11:33:34 +0000 UTC" deadline="2028-06-14 07:16:07.50699313 +0000 UTC"
Sep 15 11:38:34.653481 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:34.653434    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15307h37m32.85356527s"
Sep 15 11:38:34.987287 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:34.987258    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed568cb_25cc_452a_a618_ef7b524de14a.slice/crio-6d29589b05bf8cccb517115f50dd63c9c0a45d4d7a6f7d2d6b264f79afb9512f WatchSource:0}: Error finding container 6d29589b05bf8cccb517115f50dd63c9c0a45d4d7a6f7d2d6b264f79afb9512f: Status 404 returned error can't find the container with id 6d29589b05bf8cccb517115f50dd63c9c0a45d4d7a6f7d2d6b264f79afb9512f
Sep 15 11:38:34.988969 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:34.988947    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665a9fc2_c525_47c2_9a83_9e1bd3af8ab0.slice/crio-b8bddc8894087c2b747fd0a21643740d3627a98b7d0d5daa8d54b4cb92125a98 WatchSource:0}: Error finding container b8bddc8894087c2b747fd0a21643740d3627a98b7d0d5daa8d54b4cb92125a98: Status 404 returned error can't find the container with id b8bddc8894087c2b747fd0a21643740d3627a98b7d0d5daa8d54b4cb92125a98
Sep 15 11:38:35.009482 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:35.009459    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc213c82d_7bda_4c00_a867_059ab500ca55.slice/crio-ee784b7a0138719a8041ae09c71d53af427ac26e4f83a1df0f59c1da8e49b2b4 WatchSource:0}: Error finding container ee784b7a0138719a8041ae09c71d53af427ac26e4f83a1df0f59c1da8e49b2b4: Status 404 returned error can't find the container with id ee784b7a0138719a8041ae09c71d53af427ac26e4f83a1df0f59c1da8e49b2b4
Sep 15 11:38:35.011495 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:35.011468    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc598d7_b2ab_457d_a6e1_9ae73188f8e5.slice/crio-5f4d395502cd59efbad4a258ff865a84d9e1c8e29959cced108180181f6856a5 WatchSource:0}: Error finding container 5f4d395502cd59efbad4a258ff865a84d9e1c8e29959cced108180181f6856a5: Status 404 returned error can't find the container with id 5f4d395502cd59efbad4a258ff865a84d9e1c8e29959cced108180181f6856a5
Sep 15 11:38:35.012831 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:35.012718    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0712213d_ab9e_4d03_ba6f_cbc70b17dcdb.slice/crio-a438838584c64d8a746011d6624e9e1f5cb38fdb0c5e67c29675b1aae29eee8d WatchSource:0}: Error finding container a438838584c64d8a746011d6624e9e1f5cb38fdb0c5e67c29675b1aae29eee8d: Status 404 returned error can't find the container with id a438838584c64d8a746011d6624e9e1f5cb38fdb0c5e67c29675b1aae29eee8d
Sep 15 11:38:35.015260 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:35.015213    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00a7cd2_9093_4f86_b40b_1b29b8f6d061.slice/crio-3bfe83d8eace57e5d35e64a3863faeb7be73cf4630a05f762ac15d36eb92c4e3 WatchSource:0}: Error finding container 3bfe83d8eace57e5d35e64a3863faeb7be73cf4630a05f762ac15d36eb92c4e3: Status 404 returned error can't find the container with id 3bfe83d8eace57e5d35e64a3863faeb7be73cf4630a05f762ac15d36eb92c4e3
Sep 15 11:38:35.016639 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:35.016603    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5467c994_9999_4b50_880d_538763991a9e.slice/crio-5a02acdbed6f52426a74af377295d67dc773e2811007e0a3bff861be7407974c WatchSource:0}: Error finding container 5a02acdbed6f52426a74af377295d67dc773e2811007e0a3bff861be7407974c: Status 404 returned error can't find the container with id 5a02acdbed6f52426a74af377295d67dc773e2811007e0a3bff861be7407974c
Sep 15 11:38:35.018029 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:35.017883    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a407e8_ad7e_4047_aff1_31aa68ac6021.slice/crio-085a06b9dd5073ab2d46301494d5349696c0fbbb830b1be7b474f9921f04c807 WatchSource:0}: Error finding container 085a06b9dd5073ab2d46301494d5349696c0fbbb830b1be7b474f9921f04c807: Status 404 returned error can't find the container with id 085a06b9dd5073ab2d46301494d5349696c0fbbb830b1be7b474f9921f04c807
Sep 15 11:38:35.019321 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:38:35.019206    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81aebea3_e5e1_4326_bfb2_1bec4e195109.slice/crio-740078971a98d16103d1846deeb812b47c406edad395e995c916a707e0589f8f WatchSource:0}: Error finding container 740078971a98d16103d1846deeb812b47c406edad395e995c916a707e0589f8f: Status 404 returned error can't find the container with id 740078971a98d16103d1846deeb812b47c406edad395e995c916a707e0589f8f
Sep 15 11:38:35.235463 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.235383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk99l\" (UniqueName: \"kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l\") pod \"network-check-target-dvh4t\" (UID: \"b14e7c6d-9418-49ca-81f8-2b81ed8bce7b\") " pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:35.235463 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.235430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:35.235650 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:35.235562    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:38:35.235650 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:35.235585    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:38:35.235650 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:35.235612    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:38:35.235777 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:35.235676    2580 projected.go:196] Error preparing data for projected volume kube-api-access-tk99l for pod openshift-network-diagnostics/network-check-target-dvh4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:38:35.235777 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:35.235620    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret podName:ec648b53-d8f0-43aa-82d3-5ae9212975d6 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:37.235606814 +0000 UTC m=+6.016178181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret") pod "global-pull-secret-syncer-6tnl8" (UID: "ec648b53-d8f0-43aa-82d3-5ae9212975d6") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:38:35.235777 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:35.235737    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l podName:b14e7c6d-9418-49ca-81f8-2b81ed8bce7b nodeName:}" failed. No retries permitted until 2026-09-15 11:38:37.235719514 +0000 UTC m=+6.016290901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk99l" (UniqueName: "kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l") pod "network-check-target-dvh4t" (UID: "b14e7c6d-9418-49ca-81f8-2b81ed8bce7b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:38:35.336010 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.335982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs\") pod \"network-metrics-daemon-rxkvf\" (UID: \"1263d93a-e6b5-497d-a022-f24619717fa9\") " pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:35.336148 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:35.336128    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:38:35.336202 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:35.336194    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs podName:1263d93a-e6b5-497d-a022-f24619717fa9 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:37.336179065 +0000 UTC m=+6.116750438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs") pod "network-metrics-daemon-rxkvf" (UID: "1263d93a-e6b5-497d-a022-f24619717fa9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:38:35.654547 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.654426    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 11:33:34 +0000 UTC" deadline="2028-04-17 05:23:48.44317583 +0000 UTC"
Sep 15 11:38:35.654547 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.654467    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13913h45m12.788714148s"
Sep 15 11:38:35.727403 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.727368    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:35.727568 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:35.727527    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:38:35.727990 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.727970    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:35.728105 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:35.728083    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:38:35.728175 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.728154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:35.728238 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:35.728224    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:38:35.747050 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.747018    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hd8pm" event={"ID":"d00a7cd2-9093-4f86-b40b-1b29b8f6d061","Type":"ContainerStarted","Data":"3bfe83d8eace57e5d35e64a3863faeb7be73cf4630a05f762ac15d36eb92c4e3"}
Sep 15 11:38:35.766249 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.766040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ptgh5" event={"ID":"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb","Type":"ContainerStarted","Data":"a438838584c64d8a746011d6624e9e1f5cb38fdb0c5e67c29675b1aae29eee8d"}
Sep 15 11:38:35.772732 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.772613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b899l" event={"ID":"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5","Type":"ContainerStarted","Data":"5f4d395502cd59efbad4a258ff865a84d9e1c8e29959cced108180181f6856a5"}
Sep 15 11:38:35.777934 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.777905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vnm5x" event={"ID":"c213c82d-7bda-4c00-a867-059ab500ca55","Type":"ContainerStarted","Data":"ee784b7a0138719a8041ae09c71d53af427ac26e4f83a1df0f59c1da8e49b2b4"}
Sep 15 11:38:35.796853 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.796797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5" event={"ID":"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0","Type":"ContainerStarted","Data":"b8bddc8894087c2b747fd0a21643740d3627a98b7d0d5daa8d54b4cb92125a98"}
Sep 15 11:38:35.802993 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.802965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pdj4g" event={"ID":"3ed568cb-25cc-452a-a618-ef7b524de14a","Type":"ContainerStarted","Data":"6d29589b05bf8cccb517115f50dd63c9c0a45d4d7a6f7d2d6b264f79afb9512f"}
Sep 15 11:38:35.807390 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.807359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-211.ec2.internal" event={"ID":"acc80c84a62ea6b15d405529c0cb63ed","Type":"ContainerStarted","Data":"9204176ad9597fc83298b4a50451da9c18da8782541816c9d7ca4d8d77a85952"}
Sep 15 11:38:35.822137 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.822104    2580 generic.go:356] "Generic (PLEG): container finished" podID="883a21e7077324fc24e91cbc981bfd76" containerID="87f593e7c910aa27a62f94b1f6b70ab0b7d9f1c4563b5f4ea3ea31738259f33e" exitCode=0
Sep 15 11:38:35.822263 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.822200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal" event={"ID":"883a21e7077324fc24e91cbc981bfd76","Type":"ContainerDied","Data":"87f593e7c910aa27a62f94b1f6b70ab0b7d9f1c4563b5f4ea3ea31738259f33e"}
Sep 15 11:38:35.830582 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.830553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hr2q4" event={"ID":"81aebea3-e5e1-4326-bfb2-1bec4e195109","Type":"ContainerStarted","Data":"740078971a98d16103d1846deeb812b47c406edad395e995c916a707e0589f8f"}
Sep 15 11:38:35.842060 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.841903    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9" event={"ID":"27a407e8-ad7e-4047-aff1-31aa68ac6021","Type":"ContainerStarted","Data":"085a06b9dd5073ab2d46301494d5349696c0fbbb830b1be7b474f9921f04c807"}
Sep 15 11:38:35.855072 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.852484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wjqmg" event={"ID":"5467c994-9999-4b50-880d-538763991a9e","Type":"ContainerStarted","Data":"5a02acdbed6f52426a74af377295d67dc773e2811007e0a3bff861be7407974c"}
Sep 15 11:38:35.859018 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:35.858942    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-211.ec2.internal" podStartSLOduration=2.858923805 podStartE2EDuration="2.858923805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:35.829344306 +0000 UTC m=+4.609915698" watchObservedRunningTime="2026-09-15 11:38:35.858923805 +0000 UTC m=+4.639495198"
Sep 15 11:38:36.894686 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:36.894149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal" event={"ID":"883a21e7077324fc24e91cbc981bfd76","Type":"ContainerStarted","Data":"b4aadd40d3587a4bc2c8a348307282f0ce75bde499d5639ff18b0fed504c1251"}
Sep 15 11:38:37.258169 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:37.257200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk99l\" (UniqueName: \"kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l\") pod \"network-check-target-dvh4t\" (UID: \"b14e7c6d-9418-49ca-81f8-2b81ed8bce7b\") " pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:37.258169 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:37.257264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:37.258169 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:37.257427    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:38:37.258169 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:37.257497    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret podName:ec648b53-d8f0-43aa-82d3-5ae9212975d6 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:41.257475767 +0000 UTC m=+10.038047141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret") pod "global-pull-secret-syncer-6tnl8" (UID: "ec648b53-d8f0-43aa-82d3-5ae9212975d6") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:38:37.258169 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:37.257965    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:38:37.258169 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:37.257991    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:38:37.258169 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:37.258006    2580 projected.go:196] Error preparing data for projected volume kube-api-access-tk99l for pod openshift-network-diagnostics/network-check-target-dvh4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:38:37.258169 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:37.258057    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l podName:b14e7c6d-9418-49ca-81f8-2b81ed8bce7b nodeName:}" failed. No retries permitted until 2026-09-15 11:38:41.258042374 +0000 UTC m=+10.038613747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk99l" (UniqueName: "kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l") pod "network-check-target-dvh4t" (UID: "b14e7c6d-9418-49ca-81f8-2b81ed8bce7b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:38:37.357764 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:37.357635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs\") pod \"network-metrics-daemon-rxkvf\" (UID: \"1263d93a-e6b5-497d-a022-f24619717fa9\") " pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:37.358020 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:37.357809    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:38:37.358020 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:37.357917    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs podName:1263d93a-e6b5-497d-a022-f24619717fa9 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:41.357893361 +0000 UTC m=+10.138464745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs") pod "network-metrics-daemon-rxkvf" (UID: "1263d93a-e6b5-497d-a022-f24619717fa9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:38:37.727420 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:37.727308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:37.727589 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:37.727439    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:38:37.727881 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:37.727858    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:37.728007 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:37.727983    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:38:37.728082 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:37.728057    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:37.728156 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:37.728131    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:38:39.728703 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:39.727816    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:39.728703 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:39.727984    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:38:39.728703 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:39.728406    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:39.728703 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:39.728513    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:38:39.728703 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:39.728572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:39.728703 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:39.728646    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:38:41.290281 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:41.290135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk99l\" (UniqueName: \"kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l\") pod \"network-check-target-dvh4t\" (UID: \"b14e7c6d-9418-49ca-81f8-2b81ed8bce7b\") " pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:41.290281 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:41.290216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:41.290771 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:41.290335    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:38:41.290771 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:41.290373    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:38:41.290771 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:41.290389    2580 projected.go:196] Error preparing data for projected volume kube-api-access-tk99l for pod openshift-network-diagnostics/network-check-target-dvh4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:38:41.290771 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:41.290456    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l podName:b14e7c6d-9418-49ca-81f8-2b81ed8bce7b nodeName:}" failed. No retries permitted until 2026-09-15 11:38:49.290437851 +0000 UTC m=+18.071009223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk99l" (UniqueName: "kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l") pod "network-check-target-dvh4t" (UID: "b14e7c6d-9418-49ca-81f8-2b81ed8bce7b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:38:41.290771 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:41.290345    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:38:41.290771 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:41.290560    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret podName:ec648b53-d8f0-43aa-82d3-5ae9212975d6 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:49.290530689 +0000 UTC m=+18.071102063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret") pod "global-pull-secret-syncer-6tnl8" (UID: "ec648b53-d8f0-43aa-82d3-5ae9212975d6") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:38:41.391444 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:41.391403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs\") pod \"network-metrics-daemon-rxkvf\" (UID: \"1263d93a-e6b5-497d-a022-f24619717fa9\") " pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:41.391760 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:41.391540    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:38:41.391760 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:41.391616    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs podName:1263d93a-e6b5-497d-a022-f24619717fa9 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:49.391597061 +0000 UTC m=+18.172168429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs") pod "network-metrics-daemon-rxkvf" (UID: "1263d93a-e6b5-497d-a022-f24619717fa9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:38:41.726528 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:41.726012    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:41.726528 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:41.726057    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:41.726528 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:41.726012    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:41.726528 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:41.726146    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:38:41.726528 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:41.726198    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:38:41.726528 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:41.726312    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:38:43.724693 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:43.724600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:43.724693 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:43.724645    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:43.725149 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:43.724600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:43.725149 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:43.724734    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:38:43.725149 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:43.724811    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:38:43.725149 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:43.724890    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:38:45.725110 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:45.725077    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:45.725491 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:45.725132    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:45.725491 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:45.725084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:45.725491 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:45.725222    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:38:45.725491 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:45.725339    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:38:45.725491 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:45.725444    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:38:47.724544 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:47.724509    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:47.724972 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:47.724509    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:47.724972 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:47.724637    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:38:47.724972 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:47.724706    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:38:47.724972 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:47.724517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:47.724972 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:47.724806    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:38:49.352282 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:49.352196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk99l\" (UniqueName: \"kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l\") pod \"network-check-target-dvh4t\" (UID: \"b14e7c6d-9418-49ca-81f8-2b81ed8bce7b\") " pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:49.352282 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:49.352249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:49.352713 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:49.352376    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:38:49.352713 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:49.352402    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:38:49.352713 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:49.352381    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:38:49.352713 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:49.352416    2580 projected.go:196] Error preparing data for projected volume kube-api-access-tk99l for pod openshift-network-diagnostics/network-check-target-dvh4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:38:49.352713 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:49.352466    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret podName:ec648b53-d8f0-43aa-82d3-5ae9212975d6 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:05.352448083 +0000 UTC m=+34.133019453 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret") pod "global-pull-secret-syncer-6tnl8" (UID: "ec648b53-d8f0-43aa-82d3-5ae9212975d6") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:38:49.352713 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:49.352484    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l podName:b14e7c6d-9418-49ca-81f8-2b81ed8bce7b nodeName:}" failed. No retries permitted until 2026-09-15 11:39:05.352474393 +0000 UTC m=+34.133045766 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk99l" (UniqueName: "kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l") pod "network-check-target-dvh4t" (UID: "b14e7c6d-9418-49ca-81f8-2b81ed8bce7b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:38:49.452778 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:49.452751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs\") pod \"network-metrics-daemon-rxkvf\" (UID: \"1263d93a-e6b5-497d-a022-f24619717fa9\") " pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:49.452952 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:49.452864    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:38:49.452952 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:49.452946    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs podName:1263d93a-e6b5-497d-a022-f24619717fa9 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:05.452928105 +0000 UTC m=+34.233499472 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs") pod "network-metrics-daemon-rxkvf" (UID: "1263d93a-e6b5-497d-a022-f24619717fa9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:38:49.724859 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:49.724823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:49.725051 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:49.724823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:49.725051 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:49.724965    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:38:49.725166 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:49.725047    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:38:49.725166 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:49.724823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:49.725228 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:49.725172    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:38:51.727678 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:51.727643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:51.728116 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:51.727747    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:51.728116 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:51.727760    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:38:51.728116 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:51.727850    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:38:51.728116 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:51.727928    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:51.728116 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:51.728050    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:38:53.725338 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.725049    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:53.726283 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:53.725371    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:38:53.726283 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.725079    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:53.726283 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:53.725467    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:38:53.726283 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.725051    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:53.726283 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:53.725566    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:38:53.941067 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.940887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9" event={"ID":"27a407e8-ad7e-4047-aff1-31aa68ac6021","Type":"ContainerStarted","Data":"494506e2ef5bfac1d51c600f13fc28c9a6e34127468186a3caae62cdb3af582d"}
Sep 15 11:38:53.942319 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.942294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wjqmg" event={"ID":"5467c994-9999-4b50-880d-538763991a9e","Type":"ContainerStarted","Data":"5377c975c9a2f41f3642a89f7022e762c1ea504fa4e8da1e1a6ba7090e1cdead"}
Sep 15 11:38:53.946627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.946596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hd8pm" event={"ID":"d00a7cd2-9093-4f86-b40b-1b29b8f6d061","Type":"ContainerStarted","Data":"8b1d59afd0cebd0bc5ea1033bf39dab85749696553f6d6a39baaea0460a9984e"}
Sep 15 11:38:53.948333 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.948305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ptgh5" event={"ID":"0712213d-ab9e-4d03-ba6f-cbc70b17dcdb","Type":"ContainerStarted","Data":"c4b7a4344dfb952aa267b9a128b50318102f492a9b54c740fabb2f09c4e01a24"}
Sep 15 11:38:53.949893 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.949857    2580 generic.go:356] "Generic (PLEG): container finished" podID="4cc598d7-b2ab-457d-a6e1-9ae73188f8e5" containerID="31cf5e04bccb295789b1f3a48d8cafbc864d40a6c7e46ccdf036b72c3bc9aa70" exitCode=0
Sep 15 11:38:53.950059 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.949905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b899l" event={"ID":"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5","Type":"ContainerDied","Data":"31cf5e04bccb295789b1f3a48d8cafbc864d40a6c7e46ccdf036b72c3bc9aa70"}
Sep 15 11:38:53.953783 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.953759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vnm5x" event={"ID":"c213c82d-7bda-4c00-a867-059ab500ca55","Type":"ContainerStarted","Data":"f4b470950ae07272a8d247d052b1294e021043be0d4e6c8230aa89a72788fd52"}
Sep 15 11:38:53.957529 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.957507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5" event={"ID":"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0","Type":"ContainerStarted","Data":"2e6fc4015fecc3b0b17e2d979f27761007b08b7279c37ab5100b85f464a65099"}
Sep 15 11:38:53.957634 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.957567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5" event={"ID":"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0","Type":"ContainerStarted","Data":"3b0d3d93f93a1569efc8799933eea2f113168c1526a214f956f395d82c1454bc"}
Sep 15 11:38:53.957634 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.957615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5" event={"ID":"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0","Type":"ContainerStarted","Data":"ac6bec2b55a9be852c82dc26ed1230e57efde4db77c608cdc839b8db42704735"}
Sep 15 11:38:53.957750 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.957649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5" event={"ID":"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0","Type":"ContainerStarted","Data":"1d6c5ac85b7d4ec05a367ed1912134c2caf42f21cbdef8d1ae6d81f0d96245ae"}
Sep 15 11:38:53.957750 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.957689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5" event={"ID":"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0","Type":"ContainerStarted","Data":"2dd1fe611896b14034edead4510b2336d5cd450c22c8b0da1651dc9980463ba4"}
Sep 15 11:38:53.957750 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.957732    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5" event={"ID":"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0","Type":"ContainerStarted","Data":"be05045e13a4147b101fb7659c71c0778b043c58044e964e2a4e18fe4c6dfc2e"}
Sep 15 11:38:53.959029 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.958991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pdj4g" event={"ID":"3ed568cb-25cc-452a-a618-ef7b524de14a","Type":"ContainerStarted","Data":"2107fd10a7fbb489162dfcff2734ce57201db7429c2ea6323462c98f9d784c25"}
Sep 15 11:38:53.959164 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.959126    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wjqmg" podStartSLOduration=4.01995202 podStartE2EDuration="21.959112992s" totalImagesPullingTime="17.939160972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:53.958815657 +0000 UTC m=+22.739387046" watchObservedRunningTime="2026-09-15 11:38:53.959112992 +0000 UTC m=+22.739684381"
Sep 15 11:38:53.959650 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.959621    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-211.ec2.internal" podStartSLOduration=20.959611693 podStartE2EDuration="20.959611693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:36.913070683 +0000 UTC m=+5.693642076" watchObservedRunningTime="2026-09-15 11:38:53.959611693 +0000 UTC m=+22.740183081"
Sep 15 11:38:53.973608 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.973566    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hd8pm" podStartSLOduration=4.034390643 podStartE2EDuration="21.973551348s" totalImagesPullingTime="17.939160705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:53.973132798 +0000 UTC m=+22.753704179" watchObservedRunningTime="2026-09-15 11:38:53.973551348 +0000 UTC m=+22.754122737"
Sep 15 11:38:53.989365 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:53.989330    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vnm5x" podStartSLOduration=3.984001632 podStartE2EDuration="21.989316105s" totalImagesPullingTime="18.005314473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:53.989271669 +0000 UTC m=+22.769843074" watchObservedRunningTime="2026-09-15 11:38:53.989316105 +0000 UTC m=+22.769887494"
Sep 15 11:38:54.005381 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:54.005349    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ptgh5" podStartSLOduration=4.062775081 podStartE2EDuration="22.00533355s" totalImagesPullingTime="17.942558469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:54.004991308 +0000 UTC m=+22.785562695" watchObservedRunningTime="2026-09-15 11:38:54.00533355 +0000 UTC m=+22.785904957"
Sep 15 11:38:54.038750 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:54.038715    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pdj4g" podStartSLOduration=7.994094963 podStartE2EDuration="22.038702337s" totalImagesPullingTime="14.044607374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:54.038334919 +0000 UTC m=+22.818906306" watchObservedRunningTime="2026-09-15 11:38:54.038702337 +0000 UTC m=+22.819273725"
Sep 15 11:38:54.483478 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:54.483456    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 11:38:54.663792 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:54.663687    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T11:38:54.483474133Z","UUID":"8fd68c9e-863b-4f15-b87c-6793903b56c1","Handler":null,"Name":"","Endpoint":""}
Sep 15 11:38:54.666904 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:54.666864    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 11:38:54.667042 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:54.666917    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 11:38:54.963721 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:54.963637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hr2q4" event={"ID":"81aebea3-e5e1-4326-bfb2-1bec4e195109","Type":"ContainerStarted","Data":"bf9a402f7b3f2eed2925c882aa1fb202814b022c53991fa6ea626e1a985ec40f"}
Sep 15 11:38:54.967225 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:54.967193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9" event={"ID":"27a407e8-ad7e-4047-aff1-31aa68ac6021","Type":"ContainerStarted","Data":"15a7abe9f0e6bb0398844b746fdebf413fa600f8f658baa5ce184e5590972a8f"}
Sep 15 11:38:54.981234 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:54.981187    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hr2q4" podStartSLOduration=5.044250445 podStartE2EDuration="22.981168129s" totalImagesPullingTime="17.936917684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:54.981142799 +0000 UTC m=+23.761714191" watchObservedRunningTime="2026-09-15 11:38:54.981168129 +0000 UTC m=+23.761739518"
Sep 15 11:38:55.724898 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:55.724407    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:55.724898 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:55.724416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:55.724898 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:55.724542    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:55.724898 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:55.724541    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:38:55.724898 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:55.724743    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:38:55.724898 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:55.724848    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:38:55.972302 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:55.972268    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9" event={"ID":"27a407e8-ad7e-4047-aff1-31aa68ac6021","Type":"ContainerStarted","Data":"f4ed1ce90c22262e78b5dac082c9107be68b64ee62b27cb6405199cb2db7c7a9"}
Sep 15 11:38:55.976441 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:55.976403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5" event={"ID":"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0","Type":"ContainerStarted","Data":"909fb9aab6e395543188d23bea75ed9622d5b034873f1e327925e4a3a77d7a74"}
Sep 15 11:38:55.993530 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:55.993484    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqcj9" podStartSLOduration=3.747461285 podStartE2EDuration="23.993472797s" totalImagesPullingTime="20.246011512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:55.993337815 +0000 UTC m=+24.773909202" watchObservedRunningTime="2026-09-15 11:38:55.993472797 +0000 UTC m=+24.774044184"
Sep 15 11:38:57.725121 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:57.725092    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:57.725542 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:57.725092    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:57.725542 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:57.725207    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:38:57.725542 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:57.725299    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:38:57.725542 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:57.725093    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:57.725542 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:57.725398    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:38:58.896980 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:58.896818    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pdj4g"
Sep 15 11:38:58.897699 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:58.897557    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pdj4g"
Sep 15 11:38:58.985853 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:58.985776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b899l" event={"ID":"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5","Type":"ContainerStarted","Data":"38cdd220ac2a3274657dcf56aa45c99fd5e4c326c901e04f19e68e8dfdd85f57"}
Sep 15 11:38:58.989704 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:58.989661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5" event={"ID":"665a9fc2-c525-47c2-9a83-9e1bd3af8ab0","Type":"ContainerStarted","Data":"05e7d462572ec134660ebdcc732cb7b6de6639f5c0ecc7ab48a33511aac522e5"}
Sep 15 11:38:58.989919 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:58.989902    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pdj4g"
Sep 15 11:38:58.990439 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:58.990418    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pdj4g"
Sep 15 11:38:59.050740 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:59.050696    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5" podStartSLOduration=9.046757025 podStartE2EDuration="27.050680823s" totalImagesPullingTime="18.003923798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:59.04972365 +0000 UTC m=+27.830295038" watchObservedRunningTime="2026-09-15 11:38:59.050680823 +0000 UTC m=+27.831252211"
Sep 15 11:38:59.724413 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:59.724389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:38:59.724413 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:59.724403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:38:59.724627 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:59.724488    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:38:59.724627 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:59.724533    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:38:59.724627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:59.724593    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:38:59.724783 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:38:59.724716    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:38:59.994089 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:59.993974    2580 generic.go:356] "Generic (PLEG): container finished" podID="4cc598d7-b2ab-457d-a6e1-9ae73188f8e5" containerID="38cdd220ac2a3274657dcf56aa45c99fd5e4c326c901e04f19e68e8dfdd85f57" exitCode=0
Sep 15 11:38:59.994089 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:59.994075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b899l" event={"ID":"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5","Type":"ContainerDied","Data":"38cdd220ac2a3274657dcf56aa45c99fd5e4c326c901e04f19e68e8dfdd85f57"}
Sep 15 11:38:59.994657 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:59.994371    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:59.994839 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:59.994805    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:38:59.994979 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:38:59.994892    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:39:00.012908 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:00.012855    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:39:00.014360 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:00.014338    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:39:00.752519 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:00.752350    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rxkvf"]
Sep 15 11:39:00.752675 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:00.752660    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:39:00.752817 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:00.752772    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:39:00.755325 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:00.755291    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6tnl8"]
Sep 15 11:39:00.755438 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:00.755426    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:39:00.755545 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:00.755523    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:39:00.755904 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:00.755865    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dvh4t"]
Sep 15 11:39:00.755981 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:00.755968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:39:00.756058 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:00.756038    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:39:00.997997 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:00.997963    2580 generic.go:356] "Generic (PLEG): container finished" podID="4cc598d7-b2ab-457d-a6e1-9ae73188f8e5" containerID="9dc79832aedcafa962824fac11a9bdb5efcd9ab1a83aad0cf6ffcb8ec22737ec" exitCode=0
Sep 15 11:39:00.998531 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:00.998051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b899l" event={"ID":"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5","Type":"ContainerDied","Data":"9dc79832aedcafa962824fac11a9bdb5efcd9ab1a83aad0cf6ffcb8ec22737ec"}
Sep 15 11:39:02.003113 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:02.003068    2580 generic.go:356] "Generic (PLEG): container finished" podID="4cc598d7-b2ab-457d-a6e1-9ae73188f8e5" containerID="9f40c88ee45d7522719def1723211090e8595bb22b9cd68bbf1c96db1decfce4" exitCode=0
Sep 15 11:39:02.003513 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:02.003137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b899l" event={"ID":"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5","Type":"ContainerDied","Data":"9f40c88ee45d7522719def1723211090e8595bb22b9cd68bbf1c96db1decfce4"}
Sep 15 11:39:02.724564 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:02.724374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:39:02.724725 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:02.724409    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:39:02.724725 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:02.724660    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:39:02.724725 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:02.724409    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:39:02.724907 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:02.724765    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:39:02.724907 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:02.724818    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:39:04.725090 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:04.725057    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:39:04.725090 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:04.725085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:39:04.725639 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:04.725157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:39:04.725639 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:04.725177    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tnl8" podUID="ec648b53-d8f0-43aa-82d3-5ae9212975d6"
Sep 15 11:39:04.725639 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:04.725256    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvh4t" podUID="b14e7c6d-9418-49ca-81f8-2b81ed8bce7b"
Sep 15 11:39:04.725639 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:04.725366    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxkvf" podUID="1263d93a-e6b5-497d-a022-f24619717fa9"
Sep 15 11:39:05.364697 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:05.364653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk99l\" (UniqueName: \"kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l\") pod \"network-check-target-dvh4t\" (UID: \"b14e7c6d-9418-49ca-81f8-2b81ed8bce7b\") " pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:39:05.365089 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:05.364719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:39:05.365089 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:05.364854    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:39:05.365089 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:05.364858    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:39:05.365089 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:05.364901    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:39:05.365089 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:05.364916    2580 projected.go:196] Error preparing data for projected volume kube-api-access-tk99l for pod openshift-network-diagnostics/network-check-target-dvh4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:39:05.365089 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:05.364938    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret podName:ec648b53-d8f0-43aa-82d3-5ae9212975d6 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:37.364919266 +0000 UTC m=+66.145490654 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret") pod "global-pull-secret-syncer-6tnl8" (UID: "ec648b53-d8f0-43aa-82d3-5ae9212975d6") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:39:05.365089 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:05.364974    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l podName:b14e7c6d-9418-49ca-81f8-2b81ed8bce7b nodeName:}" failed. No retries permitted until 2026-09-15 11:39:37.364957293 +0000 UTC m=+66.145528659 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk99l" (UniqueName: "kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l") pod "network-check-target-dvh4t" (UID: "b14e7c6d-9418-49ca-81f8-2b81ed8bce7b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:39:05.465496 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:05.465462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs\") pod \"network-metrics-daemon-rxkvf\" (UID: \"1263d93a-e6b5-497d-a022-f24619717fa9\") " pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:39:05.465691 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:05.465606    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:39:05.465691 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:05.465669    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs podName:1263d93a-e6b5-497d-a022-f24619717fa9 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:37.465650429 +0000 UTC m=+66.246221818 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs") pod "network-metrics-daemon-rxkvf" (UID: "1263d93a-e6b5-497d-a022-f24619717fa9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:39:06.017886 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.017844    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-211.ec2.internal" event="NodeReady"
Sep 15 11:39:06.018342 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.018020    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 11:39:06.061345 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.061315    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6pcgn"]
Sep 15 11:39:06.117073 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.116997    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6pvbr"]
Sep 15 11:39:06.117220 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.117141    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:06.120794 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.120630    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 11:39:06.121622 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.121594    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hx5hb\""
Sep 15 11:39:06.121725 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.121655    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 11:39:06.132047 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.132024    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6pcgn"]
Sep 15 11:39:06.132160 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.132067    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6pvbr"]
Sep 15 11:39:06.132218 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.132169    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6pvbr"
Sep 15 11:39:06.134411 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.134390    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 11:39:06.134547 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.134526    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vqwj6\""
Sep 15 11:39:06.134646 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.134596    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 11:39:06.134856 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.134840    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 11:39:06.271663 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.271629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/657cdce3-0695-499e-af03-9e946aa5d6d5-cert\") pod \"ingress-canary-6pvbr\" (UID: \"657cdce3-0695-499e-af03-9e946aa5d6d5\") " pod="openshift-ingress-canary/ingress-canary-6pvbr"
Sep 15 11:39:06.271854 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.271699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxdqt\" (UniqueName: \"kubernetes.io/projected/657cdce3-0695-499e-af03-9e946aa5d6d5-kube-api-access-fxdqt\") pod \"ingress-canary-6pvbr\" (UID: \"657cdce3-0695-499e-af03-9e946aa5d6d5\") " pod="openshift-ingress-canary/ingress-canary-6pvbr"
Sep 15 11:39:06.271854 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.271750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhwxh\" (UniqueName: \"kubernetes.io/projected/e2ea32c9-131c-42c0-b364-7802d07ff5d8-kube-api-access-zhwxh\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:06.271854 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.271780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:06.272053 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.271841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ea32c9-131c-42c0-b364-7802d07ff5d8-config-volume\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:06.272053 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.271939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2ea32c9-131c-42c0-b364-7802d07ff5d8-tmp-dir\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:06.372668 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.372591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:06.372668 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.372628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ea32c9-131c-42c0-b364-7802d07ff5d8-config-volume\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:06.372668 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.372645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2ea32c9-131c-42c0-b364-7802d07ff5d8-tmp-dir\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:06.372668 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.372660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/657cdce3-0695-499e-af03-9e946aa5d6d5-cert\") pod \"ingress-canary-6pvbr\" (UID: \"657cdce3-0695-499e-af03-9e946aa5d6d5\") " pod="openshift-ingress-canary/ingress-canary-6pvbr"
Sep 15 11:39:06.372987 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:06.372758    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:39:06.372987 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.372797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxdqt\" (UniqueName: \"kubernetes.io/projected/657cdce3-0695-499e-af03-9e946aa5d6d5-kube-api-access-fxdqt\") pod \"ingress-canary-6pvbr\" (UID: \"657cdce3-0695-499e-af03-9e946aa5d6d5\") " pod="openshift-ingress-canary/ingress-canary-6pvbr"
Sep 15 11:39:06.372987 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:06.372829    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls podName:e2ea32c9-131c-42c0-b364-7802d07ff5d8 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:06.872807891 +0000 UTC m=+35.653379268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls") pod "dns-default-6pcgn" (UID: "e2ea32c9-131c-42c0-b364-7802d07ff5d8") : secret "dns-default-metrics-tls" not found
Sep 15 11:39:06.372987 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.372912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhwxh\" (UniqueName: \"kubernetes.io/projected/e2ea32c9-131c-42c0-b364-7802d07ff5d8-kube-api-access-zhwxh\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:06.373269 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.373080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2ea32c9-131c-42c0-b364-7802d07ff5d8-tmp-dir\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:06.373269 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.373219    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ea32c9-131c-42c0-b364-7802d07ff5d8-config-volume\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:06.376735 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.376715    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/657cdce3-0695-499e-af03-9e946aa5d6d5-cert\") pod \"ingress-canary-6pvbr\" (UID: \"657cdce3-0695-499e-af03-9e946aa5d6d5\") " pod="openshift-ingress-canary/ingress-canary-6pvbr"
Sep 15 11:39:06.382406 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.382384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhwxh\" (UniqueName: \"kubernetes.io/projected/e2ea32c9-131c-42c0-b364-7802d07ff5d8-kube-api-access-zhwxh\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:06.382509 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.382493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxdqt\" (UniqueName: \"kubernetes.io/projected/657cdce3-0695-499e-af03-9e946aa5d6d5-kube-api-access-fxdqt\") pod \"ingress-canary-6pvbr\" (UID: \"657cdce3-0695-499e-af03-9e946aa5d6d5\") " pod="openshift-ingress-canary/ingress-canary-6pvbr"
Sep 15 11:39:06.441583 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.441555    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6pvbr"
Sep 15 11:39:06.725312 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.725293    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:39:06.725465 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.725295    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:39:06.725515 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.725295    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:39:06.728701 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.728522    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 11:39:06.728701 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.728556    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dmnh2\""
Sep 15 11:39:06.728701 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.728562    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bwddj\""
Sep 15 11:39:06.728701 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.728556    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 11:39:06.728701 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.728605    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 11:39:06.728701 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.728566    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 11:39:06.749224 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.749128    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6pvbr"]
Sep 15 11:39:06.753156 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:39:06.752735    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657cdce3_0695_499e_af03_9e946aa5d6d5.slice/crio-ccc668ba8d05499e6007d17ee37aea23d8b64da24ac25c680e1fbb75a14790d1 WatchSource:0}: Error finding container ccc668ba8d05499e6007d17ee37aea23d8b64da24ac25c680e1fbb75a14790d1: Status 404 returned error can't find the container with id ccc668ba8d05499e6007d17ee37aea23d8b64da24ac25c680e1fbb75a14790d1
Sep 15 11:39:06.877043 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:06.876776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:06.877203 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:06.876939    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:39:06.877203 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:06.877164    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls podName:e2ea32c9-131c-42c0-b364-7802d07ff5d8 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:07.877148721 +0000 UTC m=+36.657720087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls") pod "dns-default-6pcgn" (UID: "e2ea32c9-131c-42c0-b364-7802d07ff5d8") : secret "dns-default-metrics-tls" not found
Sep 15 11:39:07.019076 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:07.019007    2580 generic.go:356] "Generic (PLEG): container finished" podID="4cc598d7-b2ab-457d-a6e1-9ae73188f8e5" containerID="8491bbd74278a1c3d0c8ddf7cce3f51e2312146eae2542c3535a9e8be6d2a7bc" exitCode=0
Sep 15 11:39:07.019857 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:07.019070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b899l" event={"ID":"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5","Type":"ContainerDied","Data":"8491bbd74278a1c3d0c8ddf7cce3f51e2312146eae2542c3535a9e8be6d2a7bc"}
Sep 15 11:39:07.020245 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:07.020224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6pvbr" event={"ID":"657cdce3-0695-499e-af03-9e946aa5d6d5","Type":"ContainerStarted","Data":"ccc668ba8d05499e6007d17ee37aea23d8b64da24ac25c680e1fbb75a14790d1"}
Sep 15 11:39:07.885192 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:07.885154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:07.885392 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:07.885302    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:39:07.885392 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:07.885373    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls podName:e2ea32c9-131c-42c0-b364-7802d07ff5d8 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:09.88535793 +0000 UTC m=+38.665929296 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls") pod "dns-default-6pcgn" (UID: "e2ea32c9-131c-42c0-b364-7802d07ff5d8") : secret "dns-default-metrics-tls" not found
Sep 15 11:39:08.026943 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:08.026904    2580 generic.go:356] "Generic (PLEG): container finished" podID="4cc598d7-b2ab-457d-a6e1-9ae73188f8e5" containerID="e7273a25818c00e40cd1abbf4c4719ea7d183235696cb8f250b8313fe5ec1da6" exitCode=0
Sep 15 11:39:08.027398 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:08.026982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b899l" event={"ID":"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5","Type":"ContainerDied","Data":"e7273a25818c00e40cd1abbf4c4719ea7d183235696cb8f250b8313fe5ec1da6"}
Sep 15 11:39:09.033916 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:09.033865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b899l" event={"ID":"4cc598d7-b2ab-457d-a6e1-9ae73188f8e5","Type":"ContainerStarted","Data":"6fe4c0152b2345d72754e104c55a3b967e2a5ac6c53c1c2e0320f0a055afa43a"}
Sep 15 11:39:09.057336 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:09.057289    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b899l" podStartSLOduration=8.284304726 podStartE2EDuration="37.05726944s" totalImagesPullingTime="28.772964714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:09.056279436 +0000 UTC m=+37.836850824" watchObservedRunningTime="2026-09-15 11:39:09.05726944 +0000 UTC m=+37.837840830"
Sep 15 11:39:09.902683 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:09.902651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:09.902855 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:09.902832    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:39:09.902942 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:09.902930    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls podName:e2ea32c9-131c-42c0-b364-7802d07ff5d8 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:13.90290763 +0000 UTC m=+42.683479012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls") pod "dns-default-6pcgn" (UID: "e2ea32c9-131c-42c0-b364-7802d07ff5d8") : secret "dns-default-metrics-tls" not found
Sep 15 11:39:10.037983 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:10.037909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6pvbr" event={"ID":"657cdce3-0695-499e-af03-9e946aa5d6d5","Type":"ContainerStarted","Data":"258d134a07fbd242799ab6928c1e979ab66f8466131c2f8fa6bf5bc413830c1a"}
Sep 15 11:39:10.054914 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:10.054856    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6pvbr" podStartSLOduration=1.051006521 podStartE2EDuration="4.05484241s" totalImagesPullingTime="3.003835889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:10.054446296 +0000 UTC m=+38.835017686" watchObservedRunningTime="2026-09-15 11:39:10.05484241 +0000 UTC m=+38.835413792"
Sep 15 11:39:13.932355 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:13.932315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:13.932805 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:13.932483    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:39:13.932805 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:13.932566    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls podName:e2ea32c9-131c-42c0-b364-7802d07ff5d8 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:21.932545087 +0000 UTC m=+50.713116477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls") pod "dns-default-6pcgn" (UID: "e2ea32c9-131c-42c0-b364-7802d07ff5d8") : secret "dns-default-metrics-tls" not found
Sep 15 11:39:21.981725 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:21.981675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:21.982197 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:21.981846    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:39:21.982197 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:21.981931    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls podName:e2ea32c9-131c-42c0-b364-7802d07ff5d8 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:37.981914469 +0000 UTC m=+66.762485839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls") pod "dns-default-6pcgn" (UID: "e2ea32c9-131c-42c0-b364-7802d07ff5d8") : secret "dns-default-metrics-tls" not found
Sep 15 11:39:32.013249 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.013222    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nqs5"
Sep 15 11:39:32.156163 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.156127    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8"]
Sep 15 11:39:32.158791 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.158775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8"
Sep 15 11:39:32.161022 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.160965    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 15 11:39:32.161141 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.161064    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 15 11:39:32.161141 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.161070    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 15 11:39:32.161257 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.161166    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9m5zz\""
Sep 15 11:39:32.161257 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.161179    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:39:32.170558 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.170540    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb"]
Sep 15 11:39:32.172159 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.172144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb"
Sep 15 11:39:32.174682 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.174650    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:39:32.175043 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.174936    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 15 11:39:32.175043 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.174948    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-z9lmb\""
Sep 15 11:39:32.175191 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.175154    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8"]
Sep 15 11:39:32.175251 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.175218    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 15 11:39:32.175412 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.175395    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 15 11:39:32.184514 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.184488    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb"]
Sep 15 11:39:32.253051 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.253024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f93710b3-c313-447e-82aa-6a38090ddd60-serving-cert\") pod \"service-ca-operator-cf7c776b7-5d8m8\" (UID: \"f93710b3-c313-447e-82aa-6a38090ddd60\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8"
Sep 15 11:39:32.253172 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.253069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f93710b3-c313-447e-82aa-6a38090ddd60-config\") pod \"service-ca-operator-cf7c776b7-5d8m8\" (UID: \"f93710b3-c313-447e-82aa-6a38090ddd60\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8"
Sep 15 11:39:32.253172 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.253096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c462336-f677-41ac-94cb-a3e83fd57a3e-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-v76nb\" (UID: \"4c462336-f677-41ac-94cb-a3e83fd57a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb"
Sep 15 11:39:32.253172 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.253144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c462336-f677-41ac-94cb-a3e83fd57a3e-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-v76nb\" (UID: \"4c462336-f677-41ac-94cb-a3e83fd57a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb"
Sep 15 11:39:32.253290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.253194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75lnc\" (UniqueName: \"kubernetes.io/projected/f93710b3-c313-447e-82aa-6a38090ddd60-kube-api-access-75lnc\") pod \"service-ca-operator-cf7c776b7-5d8m8\" (UID: \"f93710b3-c313-447e-82aa-6a38090ddd60\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8"
Sep 15 11:39:32.253290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.253214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2bvs\" (UniqueName: \"kubernetes.io/projected/4c462336-f677-41ac-94cb-a3e83fd57a3e-kube-api-access-v2bvs\") pod \"kube-storage-version-migrator-operator-6c8f84f897-v76nb\" (UID: \"4c462336-f677-41ac-94cb-a3e83fd57a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb"
Sep 15 11:39:32.267025 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.266970    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f86f84bb8-8lcr2"]
Sep 15 11:39:32.268924 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.268910    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.272318 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.272300    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 11:39:32.272473 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.272456    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 11:39:32.272949 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.272926    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 11:39:32.273186 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.273174    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gswmg\""
Sep 15 11:39:32.281802 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.280615    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 11:39:32.285064 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.285025    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f86f84bb8-8lcr2"]
Sep 15 11:39:32.353878 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.353853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75lnc\" (UniqueName: \"kubernetes.io/projected/f93710b3-c313-447e-82aa-6a38090ddd60-kube-api-access-75lnc\") pod \"service-ca-operator-cf7c776b7-5d8m8\" (UID: \"f93710b3-c313-447e-82aa-6a38090ddd60\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8"
Sep 15 11:39:32.353975 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.353902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2bvs\" (UniqueName: \"kubernetes.io/projected/4c462336-f677-41ac-94cb-a3e83fd57a3e-kube-api-access-v2bvs\") pod \"kube-storage-version-migrator-operator-6c8f84f897-v76nb\" (UID: \"4c462336-f677-41ac-94cb-a3e83fd57a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb"
Sep 15 11:39:32.353975 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.353922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4fe54a34-4168-4be7-ba3d-d8230993b85c-image-registry-private-configuration\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.353975 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.353939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4fe54a34-4168-4be7-ba3d-d8230993b85c-ca-trust-extracted\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.354093 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.354050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-bound-sa-token\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.354133 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.354110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f93710b3-c313-447e-82aa-6a38090ddd60-serving-cert\") pod \"service-ca-operator-cf7c776b7-5d8m8\" (UID: \"f93710b3-c313-447e-82aa-6a38090ddd60\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8"
Sep 15 11:39:32.354168 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.354146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4fe54a34-4168-4be7-ba3d-d8230993b85c-trusted-ca\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.354210 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.354174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4fe54a34-4168-4be7-ba3d-d8230993b85c-installation-pull-secrets\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.354210 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.354195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxgk8\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-kube-api-access-nxgk8\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.354284 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.354235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f93710b3-c313-447e-82aa-6a38090ddd60-config\") pod \"service-ca-operator-cf7c776b7-5d8m8\" (UID: \"f93710b3-c313-447e-82aa-6a38090ddd60\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8"
Sep 15 11:39:32.354284 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.354251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-certificates\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.354347 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.354282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c462336-f677-41ac-94cb-a3e83fd57a3e-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-v76nb\" (UID: \"4c462336-f677-41ac-94cb-a3e83fd57a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb"
Sep 15 11:39:32.354347 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.354307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c462336-f677-41ac-94cb-a3e83fd57a3e-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-v76nb\" (UID: \"4c462336-f677-41ac-94cb-a3e83fd57a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb"
Sep 15 11:39:32.354347 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.354330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.354775 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.354749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f93710b3-c313-447e-82aa-6a38090ddd60-config\") pod \"service-ca-operator-cf7c776b7-5d8m8\" (UID: \"f93710b3-c313-447e-82aa-6a38090ddd60\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8"
Sep 15 11:39:32.354775 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.354775    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c462336-f677-41ac-94cb-a3e83fd57a3e-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-v76nb\" (UID: \"4c462336-f677-41ac-94cb-a3e83fd57a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb"
Sep 15 11:39:32.357630 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.357605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c462336-f677-41ac-94cb-a3e83fd57a3e-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-v76nb\" (UID: \"4c462336-f677-41ac-94cb-a3e83fd57a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb"
Sep 15 11:39:32.357710 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.357624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f93710b3-c313-447e-82aa-6a38090ddd60-serving-cert\") pod \"service-ca-operator-cf7c776b7-5d8m8\" (UID: \"f93710b3-c313-447e-82aa-6a38090ddd60\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8"
Sep 15 11:39:32.367631 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.367611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75lnc\" (UniqueName: \"kubernetes.io/projected/f93710b3-c313-447e-82aa-6a38090ddd60-kube-api-access-75lnc\") pod \"service-ca-operator-cf7c776b7-5d8m8\" (UID: \"f93710b3-c313-447e-82aa-6a38090ddd60\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8"
Sep 15 11:39:32.368308 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.368288    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-phssv"]
Sep 15 11:39:32.370716 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.370685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2bvs\" (UniqueName: \"kubernetes.io/projected/4c462336-f677-41ac-94cb-a3e83fd57a3e-kube-api-access-v2bvs\") pod \"kube-storage-version-migrator-operator-6c8f84f897-v76nb\" (UID: \"4c462336-f677-41ac-94cb-a3e83fd57a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb"
Sep 15 11:39:32.370959 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.370940    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-phssv"
Sep 15 11:39:32.373345 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.373327    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hdzcd\""
Sep 15 11:39:32.379441 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.379419    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-phssv"]
Sep 15 11:39:32.454820 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.454783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-bound-sa-token\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.454989 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.454834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4fe54a34-4168-4be7-ba3d-d8230993b85c-trusted-ca\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.454989 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.454851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4fe54a34-4168-4be7-ba3d-d8230993b85c-installation-pull-secrets\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.454989 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.454887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxgk8\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-kube-api-access-nxgk8\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.454989 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.454922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-certificates\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.454989 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.454943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbxvt\" (UniqueName: \"kubernetes.io/projected/4ce872ad-7e91-44ed-97d4-4456236e6bba-kube-api-access-mbxvt\") pod \"network-check-source-6f8b768455-phssv\" (UID: \"4ce872ad-7e91-44ed-97d4-4456236e6bba\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-phssv"
Sep 15 11:39:32.455251 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.455080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.455251 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.455145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4fe54a34-4168-4be7-ba3d-d8230993b85c-image-registry-private-configuration\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.455251 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.455176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4fe54a34-4168-4be7-ba3d-d8230993b85c-ca-trust-extracted\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.455251 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:32.455206    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:39:32.455251 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:32.455223    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f86f84bb8-8lcr2: secret "image-registry-tls" not found
Sep 15 11:39:32.455493 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:32.455269    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls podName:4fe54a34-4168-4be7-ba3d-d8230993b85c nodeName:}" failed. No retries permitted until 2026-09-15 11:39:32.955255413 +0000 UTC m=+61.735826779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls") pod "image-registry-5f86f84bb8-8lcr2" (UID: "4fe54a34-4168-4be7-ba3d-d8230993b85c") : secret "image-registry-tls" not found
Sep 15 11:39:32.455569 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.455552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4fe54a34-4168-4be7-ba3d-d8230993b85c-ca-trust-extracted\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.455617 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.455580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-certificates\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.455822 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.455804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4fe54a34-4168-4be7-ba3d-d8230993b85c-trusted-ca\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.457858 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.457830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4fe54a34-4168-4be7-ba3d-d8230993b85c-image-registry-private-configuration\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.457968 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.457839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4fe54a34-4168-4be7-ba3d-d8230993b85c-installation-pull-secrets\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.465208 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.465173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-bound-sa-token\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.465858 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.465840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxgk8\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-kube-api-access-nxgk8\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.467636 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.467621    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8"
Sep 15 11:39:32.482076 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.482058    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb"
Sep 15 11:39:32.556150 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.556124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbxvt\" (UniqueName: \"kubernetes.io/projected/4ce872ad-7e91-44ed-97d4-4456236e6bba-kube-api-access-mbxvt\") pod \"network-check-source-6f8b768455-phssv\" (UID: \"4ce872ad-7e91-44ed-97d4-4456236e6bba\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-phssv"
Sep 15 11:39:32.565009 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.564990    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 11:39:32.575718 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.575702    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 11:39:32.586105 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.586081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbxvt\" (UniqueName: \"kubernetes.io/projected/4ce872ad-7e91-44ed-97d4-4456236e6bba-kube-api-access-mbxvt\") pod \"network-check-source-6f8b768455-phssv\" (UID: \"4ce872ad-7e91-44ed-97d4-4456236e6bba\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-phssv"
Sep 15 11:39:32.683365 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.683334    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-phssv"
Sep 15 11:39:32.736746 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.736720    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8"]
Sep 15 11:39:32.740681 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:39:32.740654    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93710b3_c313_447e_82aa_6a38090ddd60.slice/crio-e1b1d17c17a7d34ba4c22e0475b5e34f9ed9bba56999a86964e45287ec70c826 WatchSource:0}: Error finding container e1b1d17c17a7d34ba4c22e0475b5e34f9ed9bba56999a86964e45287ec70c826: Status 404 returned error can't find the container with id e1b1d17c17a7d34ba4c22e0475b5e34f9ed9bba56999a86964e45287ec70c826
Sep 15 11:39:32.757995 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.757973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb"]
Sep 15 11:39:32.760493 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:39:32.760473    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c462336_f677_41ac_94cb_a3e83fd57a3e.slice/crio-7165ac41ce750357d691b945f6f78e24e333030516f0c8c29f6a1ee370fd4763 WatchSource:0}: Error finding container 7165ac41ce750357d691b945f6f78e24e333030516f0c8c29f6a1ee370fd4763: Status 404 returned error can't find the container with id 7165ac41ce750357d691b945f6f78e24e333030516f0c8c29f6a1ee370fd4763
Sep 15 11:39:32.948280 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.948249    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-phssv"]
Sep 15 11:39:32.951071 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:39:32.951048    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce872ad_7e91_44ed_97d4_4456236e6bba.slice/crio-0be3afc759c9913e0f4f1821bf020b4977cbb2db6a4af88266602bd312480fa9 WatchSource:0}: Error finding container 0be3afc759c9913e0f4f1821bf020b4977cbb2db6a4af88266602bd312480fa9: Status 404 returned error can't find the container with id 0be3afc759c9913e0f4f1821bf020b4977cbb2db6a4af88266602bd312480fa9
Sep 15 11:39:32.960509 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:32.960489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:32.960644 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:32.960629    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:39:32.960687 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:32.960646    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f86f84bb8-8lcr2: secret "image-registry-tls" not found
Sep 15 11:39:32.960723 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:32.960689    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls podName:4fe54a34-4168-4be7-ba3d-d8230993b85c nodeName:}" failed. No retries permitted until 2026-09-15 11:39:33.960675633 +0000 UTC m=+62.741246999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls") pod "image-registry-5f86f84bb8-8lcr2" (UID: "4fe54a34-4168-4be7-ba3d-d8230993b85c") : secret "image-registry-tls" not found
Sep 15 11:39:33.089205 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:33.089173    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb" event={"ID":"4c462336-f677-41ac-94cb-a3e83fd57a3e","Type":"ContainerStarted","Data":"7165ac41ce750357d691b945f6f78e24e333030516f0c8c29f6a1ee370fd4763"}
Sep 15 11:39:33.090207 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:33.090180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8" event={"ID":"f93710b3-c313-447e-82aa-6a38090ddd60","Type":"ContainerStarted","Data":"e1b1d17c17a7d34ba4c22e0475b5e34f9ed9bba56999a86964e45287ec70c826"}
Sep 15 11:39:33.091284 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:33.091263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-phssv" event={"ID":"4ce872ad-7e91-44ed-97d4-4456236e6bba","Type":"ContainerStarted","Data":"0be3afc759c9913e0f4f1821bf020b4977cbb2db6a4af88266602bd312480fa9"}
Sep 15 11:39:33.969585 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:33.969546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:33.969784 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:33.969720    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:39:33.969784 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:33.969743    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f86f84bb8-8lcr2: secret "image-registry-tls" not found
Sep 15 11:39:33.969941 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:33.969811    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls podName:4fe54a34-4168-4be7-ba3d-d8230993b85c nodeName:}" failed. No retries permitted until 2026-09-15 11:39:35.969787862 +0000 UTC m=+64.750359229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls") pod "image-registry-5f86f84bb8-8lcr2" (UID: "4fe54a34-4168-4be7-ba3d-d8230993b85c") : secret "image-registry-tls" not found
Sep 15 11:39:35.985966 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:35.985919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:35.986428 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:35.986104    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:39:35.986428 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:35.986130    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f86f84bb8-8lcr2: secret "image-registry-tls" not found
Sep 15 11:39:35.986428 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:35.986205    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls podName:4fe54a34-4168-4be7-ba3d-d8230993b85c nodeName:}" failed. No retries permitted until 2026-09-15 11:39:39.986184436 +0000 UTC m=+68.766755807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls") pod "image-registry-5f86f84bb8-8lcr2" (UID: "4fe54a34-4168-4be7-ba3d-d8230993b85c") : secret "image-registry-tls" not found
Sep 15 11:39:36.010558 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.010522    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j7nvb"]
Sep 15 11:39:36.013780 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.013758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.015913 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.015857    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 11:39:36.016026 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.015971    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-68t74\""
Sep 15 11:39:36.016084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.016067    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 11:39:36.016621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.016604    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 11:39:36.016752 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.016648    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 11:39:36.023649 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.023628    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j7nvb"]
Sep 15 11:39:36.086885 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.086837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.086885 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.086888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.087130 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.086946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e994d99-39e5-4133-9f60-26839d83ab39-data-volume\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.087130 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.086994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e994d99-39e5-4133-9f60-26839d83ab39-crio-socket\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.087130 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.087083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jfpg\" (UniqueName: \"kubernetes.io/projected/2e994d99-39e5-4133-9f60-26839d83ab39-kube-api-access-4jfpg\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.187809 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.187767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e994d99-39e5-4133-9f60-26839d83ab39-crio-socket\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.188033 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.187859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jfpg\" (UniqueName: \"kubernetes.io/projected/2e994d99-39e5-4133-9f60-26839d83ab39-kube-api-access-4jfpg\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.188033 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.187950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e994d99-39e5-4133-9f60-26839d83ab39-crio-socket\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.188169 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.188150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.188210 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.188181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.188246 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.188212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e994d99-39e5-4133-9f60-26839d83ab39-data-volume\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.188312 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:36.188294    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 11:39:36.188343 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:36.188308    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:39:36.188375 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:36.188369    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls podName:2e994d99-39e5-4133-9f60-26839d83ab39 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:36.688349811 +0000 UTC m=+65.468921196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7nvb" (UID: "2e994d99-39e5-4133-9f60-26839d83ab39") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:39:36.188415 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:36.188383    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm podName:2e994d99-39e5-4133-9f60-26839d83ab39 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:36.68837672 +0000 UTC m=+65.468948086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7nvb" (UID: "2e994d99-39e5-4133-9f60-26839d83ab39") : configmap "kube-rbac-proxy" not found
Sep 15 11:39:36.188507 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.188494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e994d99-39e5-4133-9f60-26839d83ab39-data-volume\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.197659 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.197633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jfpg\" (UniqueName: \"kubernetes.io/projected/2e994d99-39e5-4133-9f60-26839d83ab39-kube-api-access-4jfpg\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.693073 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.693037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.693263 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:36.693081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:36.693263 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:36.693161    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 11:39:36.693263 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:36.693232    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm podName:2e994d99-39e5-4133-9f60-26839d83ab39 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:37.693215586 +0000 UTC m=+66.473786953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7nvb" (UID: "2e994d99-39e5-4133-9f60-26839d83ab39") : configmap "kube-rbac-proxy" not found
Sep 15 11:39:36.693263 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:36.693234    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:39:36.693481 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:36.693285    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls podName:2e994d99-39e5-4133-9f60-26839d83ab39 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:37.693268624 +0000 UTC m=+66.473839997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7nvb" (UID: "2e994d99-39e5-4133-9f60-26839d83ab39") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:39:37.107884 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.107772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb" event={"ID":"4c462336-f677-41ac-94cb-a3e83fd57a3e","Type":"ContainerStarted","Data":"7c2b71aa4087049315d6647dcbcb6b2e1723428676c6b6f13ec79a92533dc3de"}
Sep 15 11:39:37.110078 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.110045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8" event={"ID":"f93710b3-c313-447e-82aa-6a38090ddd60","Type":"ContainerStarted","Data":"3b0daa309511da68f9445def8b0592547c03854e6773d3f44b1cc13124cd0d69"}
Sep 15 11:39:37.112153 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.112124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-phssv" event={"ID":"4ce872ad-7e91-44ed-97d4-4456236e6bba","Type":"ContainerStarted","Data":"fe6469b48ac690c4749c4ce3a05f4c07884f34065d32a32e44cf837cb7818f28"}
Sep 15 11:39:37.127191 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.127141    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb" podStartSLOduration=1.616468536 podStartE2EDuration="5.127123203s" totalImagesPullingTime="3.510654667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:37.125967508 +0000 UTC m=+65.906538895" watchObservedRunningTime="2026-09-15 11:39:37.127123203 +0000 UTC m=+65.907694585"
Sep 15 11:39:37.142717 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.142455    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8" podStartSLOduration=1.618489926 podStartE2EDuration="5.142439027s" totalImagesPullingTime="3.523949101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:37.142254636 +0000 UTC m=+65.922826025" watchObservedRunningTime="2026-09-15 11:39:37.142439027 +0000 UTC m=+65.923010419"
Sep 15 11:39:37.397891 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.397786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk99l\" (UniqueName: \"kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l\") pod \"network-check-target-dvh4t\" (UID: \"b14e7c6d-9418-49ca-81f8-2b81ed8bce7b\") " pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:39:37.397891 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.397837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:39:37.400463 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.400436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk99l\" (UniqueName: \"kubernetes.io/projected/b14e7c6d-9418-49ca-81f8-2b81ed8bce7b-kube-api-access-tk99l\") pod \"network-check-target-dvh4t\" (UID: \"b14e7c6d-9418-49ca-81f8-2b81ed8bce7b\") " pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:39:37.400589 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.400550    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 11:39:37.410746 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.410719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec648b53-d8f0-43aa-82d3-5ae9212975d6-original-pull-secret\") pod \"global-pull-secret-syncer-6tnl8\" (UID: \"ec648b53-d8f0-43aa-82d3-5ae9212975d6\") " pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:39:37.498645 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.498610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs\") pod \"network-metrics-daemon-rxkvf\" (UID: \"1263d93a-e6b5-497d-a022-f24619717fa9\") " pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:39:37.502901 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.502865    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 11:39:37.509243 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:37.509226    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 11:39:37.509357 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:37.509278    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs podName:1263d93a-e6b5-497d-a022-f24619717fa9 nodeName:}" failed. No retries permitted until 2026-09-15 11:40:41.50926427 +0000 UTC m=+130.289835637 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs") pod "network-metrics-daemon-rxkvf" (UID: "1263d93a-e6b5-497d-a022-f24619717fa9") : secret "metrics-daemon-secret" not found
Sep 15 11:39:37.646692 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.646653    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bwddj\""
Sep 15 11:39:37.649814 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.649758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tnl8"
Sep 15 11:39:37.654384 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.654355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:39:37.699847 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.699749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:37.700527 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.699793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:37.707206 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:37.704451    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 11:39:37.707206 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:37.704538    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm podName:2e994d99-39e5-4133-9f60-26839d83ab39 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:39.704516233 +0000 UTC m=+68.485087604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7nvb" (UID: "2e994d99-39e5-4133-9f60-26839d83ab39") : configmap "kube-rbac-proxy" not found
Sep 15 11:39:37.707206 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:37.705070    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:39:37.707206 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:37.705130    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls podName:2e994d99-39e5-4133-9f60-26839d83ab39 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:39.705107261 +0000 UTC m=+68.485678630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7nvb" (UID: "2e994d99-39e5-4133-9f60-26839d83ab39") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:39:37.964523 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.964465    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-phssv" podStartSLOduration=2.638261569 podStartE2EDuration="5.964449334s" totalImagesPullingTime="3.326187765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:37.159018484 +0000 UTC m=+65.939589874" watchObservedRunningTime="2026-09-15 11:39:37.964449334 +0000 UTC m=+66.745020733"
Sep 15 11:39:37.965483 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.965461    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6tnl8"]
Sep 15 11:39:37.968224 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:39:37.968194    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec648b53_d8f0_43aa_82d3_5ae9212975d6.slice/crio-7b7cb80c37e7230b7e3109826509ddf74a7dd8f11482988d3fcd5950cc37277c WatchSource:0}: Error finding container 7b7cb80c37e7230b7e3109826509ddf74a7dd8f11482988d3fcd5950cc37277c: Status 404 returned error can't find the container with id 7b7cb80c37e7230b7e3109826509ddf74a7dd8f11482988d3fcd5950cc37277c
Sep 15 11:39:37.971444 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:37.971418    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dvh4t"]
Sep 15 11:39:37.973523 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:39:37.973500    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14e7c6d_9418_49ca_81f8_2b81ed8bce7b.slice/crio-80bd0ea8ce37b61831ad478d93964027de68e5c4b1b5e1a1d7a62f82bc39d2b0 WatchSource:0}: Error finding container 80bd0ea8ce37b61831ad478d93964027de68e5c4b1b5e1a1d7a62f82bc39d2b0: Status 404 returned error can't find the container with id 80bd0ea8ce37b61831ad478d93964027de68e5c4b1b5e1a1d7a62f82bc39d2b0
Sep 15 11:39:38.002742 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:38.002710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:39:38.002910 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:38.002889    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:39:38.002992 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:38.002957    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls podName:e2ea32c9-131c-42c0-b364-7802d07ff5d8 nodeName:}" failed. No retries permitted until 2026-09-15 11:40:10.002937391 +0000 UTC m=+98.783508766 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls") pod "dns-default-6pcgn" (UID: "e2ea32c9-131c-42c0-b364-7802d07ff5d8") : secret "dns-default-metrics-tls" not found
Sep 15 11:39:38.115943 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:38.115909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dvh4t" event={"ID":"b14e7c6d-9418-49ca-81f8-2b81ed8bce7b","Type":"ContainerStarted","Data":"debe85c922f16b25f6789d77af664d049915165e8591ad1083b51ab5062fff61"}
Sep 15 11:39:38.116342 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:38.115970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dvh4t" event={"ID":"b14e7c6d-9418-49ca-81f8-2b81ed8bce7b","Type":"ContainerStarted","Data":"80bd0ea8ce37b61831ad478d93964027de68e5c4b1b5e1a1d7a62f82bc39d2b0"}
Sep 15 11:39:38.116342 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:38.116002    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:39:38.117113 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:38.117091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6tnl8" event={"ID":"ec648b53-d8f0-43aa-82d3-5ae9212975d6","Type":"ContainerStarted","Data":"7b7cb80c37e7230b7e3109826509ddf74a7dd8f11482988d3fcd5950cc37277c"}
Sep 15 11:39:38.132352 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:38.132313    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dvh4t" podStartSLOduration=66.132300359 podStartE2EDuration="1m6.132300359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:38.131503888 +0000 UTC m=+66.912075277" watchObservedRunningTime="2026-09-15 11:39:38.132300359 +0000 UTC m=+66.912871756"
Sep 15 11:39:38.157544 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:38.157523    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wjqmg_5467c994-9999-4b50-880d-538763991a9e/dns-node-resolver/0.log"
Sep 15 11:39:38.963347 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:38.963316    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hd8pm_d00a7cd2-9093-4f86-b40b-1b29b8f6d061/node-ca/0.log"
Sep 15 11:39:39.560759 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:39.560733    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6pvbr_657cdce3-0695-499e-af03-9e946aa5d6d5/serve-healthcheck-canary/0.log"
Sep 15 11:39:39.716987 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:39.716951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:39.717157 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:39.717002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:39.717157 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:39.717091    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 11:39:39.717288 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:39.717165    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm podName:2e994d99-39e5-4133-9f60-26839d83ab39 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:43.717144948 +0000 UTC m=+72.497716321 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7nvb" (UID: "2e994d99-39e5-4133-9f60-26839d83ab39") : configmap "kube-rbac-proxy" not found
Sep 15 11:39:39.717288 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:39.717168    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:39:39.717288 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:39.717229    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls podName:2e994d99-39e5-4133-9f60-26839d83ab39 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:43.717213385 +0000 UTC m=+72.497784767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7nvb" (UID: "2e994d99-39e5-4133-9f60-26839d83ab39") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:39:40.020728 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.020684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:40.020918 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:40.020850    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:39:40.020918 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:40.020867    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f86f84bb8-8lcr2: secret "image-registry-tls" not found
Sep 15 11:39:40.021042 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:40.020935    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls podName:4fe54a34-4168-4be7-ba3d-d8230993b85c nodeName:}" failed. No retries permitted until 2026-09-15 11:39:48.020920686 +0000 UTC m=+76.801492052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls") pod "image-registry-5f86f84bb8-8lcr2" (UID: "4fe54a34-4168-4be7-ba3d-d8230993b85c") : secret "image-registry-tls" not found
Sep 15 11:39:40.714225 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.714187    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6whtl"]
Sep 15 11:39:40.716426 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.716405    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-6whtl"
Sep 15 11:39:40.719174 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.719154    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hpp8w\""
Sep 15 11:39:40.719271 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.719153    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 15 11:39:40.719843 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.719826    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 15 11:39:40.719941 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.719847    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 15 11:39:40.720058 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.720040    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 15 11:39:40.725714 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.725692    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6whtl"]
Sep 15 11:39:40.827357 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.827323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7cqw\" (UniqueName: \"kubernetes.io/projected/e9852add-c81e-47bf-9f19-0ca92e9e8cfa-kube-api-access-p7cqw\") pod \"service-ca-568869fdc5-6whtl\" (UID: \"e9852add-c81e-47bf-9f19-0ca92e9e8cfa\") " pod="openshift-service-ca/service-ca-568869fdc5-6whtl"
Sep 15 11:39:40.827532 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.827387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e9852add-c81e-47bf-9f19-0ca92e9e8cfa-signing-cabundle\") pod \"service-ca-568869fdc5-6whtl\" (UID: \"e9852add-c81e-47bf-9f19-0ca92e9e8cfa\") " pod="openshift-service-ca/service-ca-568869fdc5-6whtl"
Sep 15 11:39:40.827532 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.827427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e9852add-c81e-47bf-9f19-0ca92e9e8cfa-signing-key\") pod \"service-ca-568869fdc5-6whtl\" (UID: \"e9852add-c81e-47bf-9f19-0ca92e9e8cfa\") " pod="openshift-service-ca/service-ca-568869fdc5-6whtl"
Sep 15 11:39:40.928656 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.928625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7cqw\" (UniqueName: \"kubernetes.io/projected/e9852add-c81e-47bf-9f19-0ca92e9e8cfa-kube-api-access-p7cqw\") pod \"service-ca-568869fdc5-6whtl\" (UID: \"e9852add-c81e-47bf-9f19-0ca92e9e8cfa\") " pod="openshift-service-ca/service-ca-568869fdc5-6whtl"
Sep 15 11:39:40.928823 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.928692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e9852add-c81e-47bf-9f19-0ca92e9e8cfa-signing-cabundle\") pod \"service-ca-568869fdc5-6whtl\" (UID: \"e9852add-c81e-47bf-9f19-0ca92e9e8cfa\") " pod="openshift-service-ca/service-ca-568869fdc5-6whtl"
Sep 15 11:39:40.928823 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.928718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e9852add-c81e-47bf-9f19-0ca92e9e8cfa-signing-key\") pod \"service-ca-568869fdc5-6whtl\" (UID: \"e9852add-c81e-47bf-9f19-0ca92e9e8cfa\") " pod="openshift-service-ca/service-ca-568869fdc5-6whtl"
Sep 15 11:39:40.929420 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.929397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e9852add-c81e-47bf-9f19-0ca92e9e8cfa-signing-cabundle\") pod \"service-ca-568869fdc5-6whtl\" (UID: \"e9852add-c81e-47bf-9f19-0ca92e9e8cfa\") " pod="openshift-service-ca/service-ca-568869fdc5-6whtl"
Sep 15 11:39:40.931468 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.931443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e9852add-c81e-47bf-9f19-0ca92e9e8cfa-signing-key\") pod \"service-ca-568869fdc5-6whtl\" (UID: \"e9852add-c81e-47bf-9f19-0ca92e9e8cfa\") " pod="openshift-service-ca/service-ca-568869fdc5-6whtl"
Sep 15 11:39:40.942211 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:40.942181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7cqw\" (UniqueName: \"kubernetes.io/projected/e9852add-c81e-47bf-9f19-0ca92e9e8cfa-kube-api-access-p7cqw\") pod \"service-ca-568869fdc5-6whtl\" (UID: \"e9852add-c81e-47bf-9f19-0ca92e9e8cfa\") " pod="openshift-service-ca/service-ca-568869fdc5-6whtl"
Sep 15 11:39:41.027211 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:41.027136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-6whtl"
Sep 15 11:39:42.106615 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:42.106536    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6whtl"]
Sep 15 11:39:42.110035 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:39:42.110008    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9852add_c81e_47bf_9f19_0ca92e9e8cfa.slice/crio-62fe80b09b9c2d9c6786acbd9aec018d8296047e33f883db84bb610f70db6f9d WatchSource:0}: Error finding container 62fe80b09b9c2d9c6786acbd9aec018d8296047e33f883db84bb610f70db6f9d: Status 404 returned error can't find the container with id 62fe80b09b9c2d9c6786acbd9aec018d8296047e33f883db84bb610f70db6f9d
Sep 15 11:39:42.130032 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:42.129993    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-6whtl" event={"ID":"e9852add-c81e-47bf-9f19-0ca92e9e8cfa","Type":"ContainerStarted","Data":"62fe80b09b9c2d9c6786acbd9aec018d8296047e33f883db84bb610f70db6f9d"}
Sep 15 11:39:42.131522 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:42.131498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6tnl8" event={"ID":"ec648b53-d8f0-43aa-82d3-5ae9212975d6","Type":"ContainerStarted","Data":"e53cb6f6a57765e4db75923cb8be53d19f3b797b47660f8c0e5777540f36741b"}
Sep 15 11:39:42.149841 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:42.149802    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6tnl8" podStartSLOduration=66.244442387 podStartE2EDuration="1m10.149787072s" totalImagesPullingTime="3.905344685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:42.14831636 +0000 UTC m=+70.928887747" watchObservedRunningTime="2026-09-15 11:39:42.149787072 +0000 UTC m=+70.930358461"
Sep 15 11:39:43.140366 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:43.140325    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-6whtl" event={"ID":"e9852add-c81e-47bf-9f19-0ca92e9e8cfa","Type":"ContainerStarted","Data":"c36773ee63f22e67b1626b2ab3e4b71fae5396358756951903e83d4d64ec5864"}
Sep 15 11:39:43.157643 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:43.157588    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-6whtl" podStartSLOduration=3.15757098 podStartE2EDuration="3.15757098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:43.15699442 +0000 UTC m=+71.937565809" watchObservedRunningTime="2026-09-15 11:39:43.15757098 +0000 UTC m=+71.938142369"
Sep 15 11:39:43.750057 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:43.750023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:43.750362 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:43.750150    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:39:43.750494 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:43.750381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:43.750494 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:43.750408    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls podName:2e994d99-39e5-4133-9f60-26839d83ab39 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:51.750387048 +0000 UTC m=+80.530958436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7nvb" (UID: "2e994d99-39e5-4133-9f60-26839d83ab39") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:39:43.750494 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:43.750453    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 11:39:43.750643 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:43.750504    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm podName:2e994d99-39e5-4133-9f60-26839d83ab39 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:51.750487345 +0000 UTC m=+80.531058717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7nvb" (UID: "2e994d99-39e5-4133-9f60-26839d83ab39") : configmap "kube-rbac-proxy" not found
Sep 15 11:39:48.083808 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:48.083770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:48.086130 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:48.086111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls\") pod \"image-registry-5f86f84bb8-8lcr2\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") " pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:48.176673 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:48.176636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:48.452847 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:48.452816    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f86f84bb8-8lcr2"]
Sep 15 11:39:48.456213 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:39:48.456175    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe54a34_4168_4be7_ba3d_d8230993b85c.slice/crio-450e14d49f57d7a4ed0bf9431bb9e162ba5880fa1d3c1c3a8a758ba8b564bc6a WatchSource:0}: Error finding container 450e14d49f57d7a4ed0bf9431bb9e162ba5880fa1d3c1c3a8a758ba8b564bc6a: Status 404 returned error can't find the container with id 450e14d49f57d7a4ed0bf9431bb9e162ba5880fa1d3c1c3a8a758ba8b564bc6a
Sep 15 11:39:49.165084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:49.165049    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2" event={"ID":"4fe54a34-4168-4be7-ba3d-d8230993b85c","Type":"ContainerStarted","Data":"addd272eaffc02c1f713119b685c2b689c3c1ea1faf381635fa55611d6141f6c"}
Sep 15 11:39:49.165547 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:49.165106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2" event={"ID":"4fe54a34-4168-4be7-ba3d-d8230993b85c","Type":"ContainerStarted","Data":"450e14d49f57d7a4ed0bf9431bb9e162ba5880fa1d3c1c3a8a758ba8b564bc6a"}
Sep 15 11:39:49.165547 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:49.165161    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:39:49.186609 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:49.186564    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2" podStartSLOduration=17.186549612 podStartE2EDuration="17.186549612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:49.18586787 +0000 UTC m=+77.966439258" watchObservedRunningTime="2026-09-15 11:39:49.186549612 +0000 UTC m=+77.967121000"
Sep 15 11:39:51.815646 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:51.815617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:51.816040 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:51.815658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:39:51.816040 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:51.815826    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:39:51.816040 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:39:51.815891    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls podName:2e994d99-39e5-4133-9f60-26839d83ab39 nodeName:}" failed. No retries permitted until 2026-09-15 11:40:07.815859052 +0000 UTC m=+96.596430418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7nvb" (UID: "2e994d99-39e5-4133-9f60-26839d83ab39") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:39:51.816303 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:39:51.816280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e994d99-39e5-4133-9f60-26839d83ab39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:40:03.056224 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.056095    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-c6p6s"]
Sep 15 11:40:03.060994 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.060971    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c6p6s"
Sep 15 11:40:03.061472 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.061452    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f86f84bb8-8lcr2"]
Sep 15 11:40:03.063174 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.063156    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-h9cbl\""
Sep 15 11:40:03.063464 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.063440    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 15 11:40:03.063649 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.063588    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 15 11:40:03.065605 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.065087    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8fbb54f84-trfst"]
Sep 15 11:40:03.066445 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.066402    2580 patch_prober.go:28] interesting pod/image-registry-5f86f84bb8-8lcr2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 11:40:03.066542 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.066453    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2" podUID="4fe54a34-4168-4be7-ba3d-d8230993b85c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 11:40:03.068134 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.068112    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"]
Sep 15 11:40:03.068247 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.068234    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8fbb54f84-trfst"
Sep 15 11:40:03.070357 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.070336    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-t2bkp\""
Sep 15 11:40:03.070667 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.070634    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 15 11:40:03.070767 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.070698    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 15 11:40:03.070767 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.070705    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 15 11:40:03.070954 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.070783    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 15 11:40:03.072091 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.072076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"
Sep 15 11:40:03.074298 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.074277    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 15 11:40:03.075832 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.075812    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-c6p6s"]
Sep 15 11:40:03.080125 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.080105    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8fbb54f84-trfst"]
Sep 15 11:40:03.092218 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.092195    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"]
Sep 15 11:40:03.156017 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.155988    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8d687b44b-k6v8j"]
Sep 15 11:40:03.159148 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.159129    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.176489 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.174911    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8d687b44b-k6v8j"]
Sep 15 11:40:03.191067 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.191042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vd54\" (UniqueName: \"kubernetes.io/projected/94211976-fca1-4277-b117-378a811c7800-kube-api-access-5vd54\") pod \"managed-serviceaccount-addon-agent-8fbb54f84-trfst\" (UID: \"94211976-fca1-4277-b117-378a811c7800\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8fbb54f84-trfst"
Sep 15 11:40:03.191164 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.191070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/83711006-89e6-4768-bae1-3e90d82459dd-klusterlet-config\") pod \"klusterlet-addon-workmgr-856c566d88-nlj88\" (UID: \"83711006-89e6-4768-bae1-3e90d82459dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"
Sep 15 11:40:03.191164 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.191095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c6p6s\" (UID: \"77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c6p6s"
Sep 15 11:40:03.191164 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.191128    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94211976-fca1-4277-b117-378a811c7800-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8fbb54f84-trfst\" (UID: \"94211976-fca1-4277-b117-378a811c7800\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8fbb54f84-trfst"
Sep 15 11:40:03.191270 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.191211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83711006-89e6-4768-bae1-3e90d82459dd-tmp\") pod \"klusterlet-addon-workmgr-856c566d88-nlj88\" (UID: \"83711006-89e6-4768-bae1-3e90d82459dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"
Sep 15 11:40:03.191270 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.191241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-c6p6s\" (UID: \"77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c6p6s"
Sep 15 11:40:03.191331 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.191269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2274g\" (UniqueName: \"kubernetes.io/projected/83711006-89e6-4768-bae1-3e90d82459dd-kube-api-access-2274g\") pod \"klusterlet-addon-workmgr-856c566d88-nlj88\" (UID: \"83711006-89e6-4768-bae1-3e90d82459dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"
Sep 15 11:40:03.291964 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.291928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-registry-tls\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.291964 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.291970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-registry-certificates\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.292216 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.291991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83711006-89e6-4768-bae1-3e90d82459dd-tmp\") pod \"klusterlet-addon-workmgr-856c566d88-nlj88\" (UID: \"83711006-89e6-4768-bae1-3e90d82459dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"
Sep 15 11:40:03.292216 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.292008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-c6p6s\" (UID: \"77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c6p6s"
Sep 15 11:40:03.292216 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.292058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2274g\" (UniqueName: \"kubernetes.io/projected/83711006-89e6-4768-bae1-3e90d82459dd-kube-api-access-2274g\") pod \"klusterlet-addon-workmgr-856c566d88-nlj88\" (UID: \"83711006-89e6-4768-bae1-3e90d82459dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"
Sep 15 11:40:03.292216 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.292118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vd54\" (UniqueName: \"kubernetes.io/projected/94211976-fca1-4277-b117-378a811c7800-kube-api-access-5vd54\") pod \"managed-serviceaccount-addon-agent-8fbb54f84-trfst\" (UID: \"94211976-fca1-4277-b117-378a811c7800\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8fbb54f84-trfst"
Sep 15 11:40:03.292216 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.292136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/83711006-89e6-4768-bae1-3e90d82459dd-klusterlet-config\") pod \"klusterlet-addon-workmgr-856c566d88-nlj88\" (UID: \"83711006-89e6-4768-bae1-3e90d82459dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"
Sep 15 11:40:03.292216 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.292161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-image-registry-private-configuration\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.292216 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.292188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zj9s\" (UniqueName: \"kubernetes.io/projected/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-kube-api-access-6zj9s\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.292216 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.292219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c6p6s\" (UID: \"77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c6p6s"
Sep 15 11:40:03.292590 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.292303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-bound-sa-token\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.292590 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.292341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-trusted-ca\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.292590 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.292375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-installation-pull-secrets\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.292590 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.292402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83711006-89e6-4768-bae1-3e90d82459dd-tmp\") pod \"klusterlet-addon-workmgr-856c566d88-nlj88\" (UID: \"83711006-89e6-4768-bae1-3e90d82459dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"
Sep 15 11:40:03.292590 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.292415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94211976-fca1-4277-b117-378a811c7800-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8fbb54f84-trfst\" (UID: \"94211976-fca1-4277-b117-378a811c7800\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8fbb54f84-trfst"
Sep 15 11:40:03.292590 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.292467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-ca-trust-extracted\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.292866 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.292681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-c6p6s\" (UID: \"77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c6p6s"
Sep 15 11:40:03.294678 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.294654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/83711006-89e6-4768-bae1-3e90d82459dd-klusterlet-config\") pod \"klusterlet-addon-workmgr-856c566d88-nlj88\" (UID: \"83711006-89e6-4768-bae1-3e90d82459dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"
Sep 15 11:40:03.294816 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.294732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c6p6s\" (UID: \"77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c6p6s"
Sep 15 11:40:03.294816 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.294760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94211976-fca1-4277-b117-378a811c7800-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8fbb54f84-trfst\" (UID: \"94211976-fca1-4277-b117-378a811c7800\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8fbb54f84-trfst"
Sep 15 11:40:03.302293 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.302254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2274g\" (UniqueName: \"kubernetes.io/projected/83711006-89e6-4768-bae1-3e90d82459dd-kube-api-access-2274g\") pod \"klusterlet-addon-workmgr-856c566d88-nlj88\" (UID: \"83711006-89e6-4768-bae1-3e90d82459dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"
Sep 15 11:40:03.302420 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.302393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vd54\" (UniqueName: \"kubernetes.io/projected/94211976-fca1-4277-b117-378a811c7800-kube-api-access-5vd54\") pod \"managed-serviceaccount-addon-agent-8fbb54f84-trfst\" (UID: \"94211976-fca1-4277-b117-378a811c7800\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8fbb54f84-trfst"
Sep 15 11:40:03.371420 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.371347    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c6p6s"
Sep 15 11:40:03.393746 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.393723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-ca-trust-extracted\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.393854 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.393770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-registry-tls\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.393854 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.393792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-registry-certificates\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.393854 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.393836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-image-registry-private-configuration\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.394047 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.393857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zj9s\" (UniqueName: \"kubernetes.io/projected/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-kube-api-access-6zj9s\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.394047 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.393919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-bound-sa-token\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.394047 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.393941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-trusted-ca\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.394189 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.394125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-installation-pull-secrets\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.394263 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.394238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-ca-trust-extracted\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.394844 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.394811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-trusted-ca\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.395030 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.395002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-registry-certificates\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.395779 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.395759    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8fbb54f84-trfst"
Sep 15 11:40:03.396867 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.396838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-image-registry-private-configuration\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.396998 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.396849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-registry-tls\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.396998 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.396947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-installation-pull-secrets\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.400941 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.400926    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"
Sep 15 11:40:03.411173 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.411124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-bound-sa-token\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.411855 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.411813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zj9s\" (UniqueName: \"kubernetes.io/projected/1cd82af1-94cb-452e-93e8-f6178b5fbe8d-kube-api-access-6zj9s\") pod \"image-registry-8d687b44b-k6v8j\" (UID: \"1cd82af1-94cb-452e-93e8-f6178b5fbe8d\") " pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.472686 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.471741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:03.642103 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.642031    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-c6p6s"]
Sep 15 11:40:03.644602 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:40:03.644576    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77711f0c_75a3_40c6_be1d_a0aaa5ca2ad2.slice/crio-619b70ce179b7b87b29f58a24ac8c33616bbf501ac9b30e838c2eb20050615f3 WatchSource:0}: Error finding container 619b70ce179b7b87b29f58a24ac8c33616bbf501ac9b30e838c2eb20050615f3: Status 404 returned error can't find the container with id 619b70ce179b7b87b29f58a24ac8c33616bbf501ac9b30e838c2eb20050615f3
Sep 15 11:40:03.674682 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.674659    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8fbb54f84-trfst"]
Sep 15 11:40:03.677021 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:40:03.677000    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94211976_fca1_4277_b117_378a811c7800.slice/crio-a32a9c0820a967cc5c7d8bc0d5b1919651f9e38ee3713d5ea55740200dc62ec5 WatchSource:0}: Error finding container a32a9c0820a967cc5c7d8bc0d5b1919651f9e38ee3713d5ea55740200dc62ec5: Status 404 returned error can't find the container with id a32a9c0820a967cc5c7d8bc0d5b1919651f9e38ee3713d5ea55740200dc62ec5
Sep 15 11:40:03.684799 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.684779    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"]
Sep 15 11:40:03.687305 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:40:03.687282    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83711006_89e6_4768_bae1_3e90d82459dd.slice/crio-688218f53ae8f18d0dac0d76d103ad7cb4118bbf478272a646c7e70dbce92353 WatchSource:0}: Error finding container 688218f53ae8f18d0dac0d76d103ad7cb4118bbf478272a646c7e70dbce92353: Status 404 returned error can't find the container with id 688218f53ae8f18d0dac0d76d103ad7cb4118bbf478272a646c7e70dbce92353
Sep 15 11:40:03.757274 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:03.757248    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8d687b44b-k6v8j"]
Sep 15 11:40:03.759830 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:40:03.759799    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd82af1_94cb_452e_93e8_f6178b5fbe8d.slice/crio-7871ae4beed9e2d113b938e6d3995d946ae8bc2bc0af155c08c62c65f27b8319 WatchSource:0}: Error finding container 7871ae4beed9e2d113b938e6d3995d946ae8bc2bc0af155c08c62c65f27b8319: Status 404 returned error can't find the container with id 7871ae4beed9e2d113b938e6d3995d946ae8bc2bc0af155c08c62c65f27b8319
Sep 15 11:40:04.212522 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:04.212489    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d687b44b-k6v8j" event={"ID":"1cd82af1-94cb-452e-93e8-f6178b5fbe8d","Type":"ContainerStarted","Data":"3b594d1bbed11f107f7144aa4bc7ff98bf81dc40d4f0872bf00844a00350684b"}
Sep 15 11:40:04.213024 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:04.212537    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d687b44b-k6v8j" event={"ID":"1cd82af1-94cb-452e-93e8-f6178b5fbe8d","Type":"ContainerStarted","Data":"7871ae4beed9e2d113b938e6d3995d946ae8bc2bc0af155c08c62c65f27b8319"}
Sep 15 11:40:04.213024 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:04.212611    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:04.213765 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:04.213742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88" event={"ID":"83711006-89e6-4768-bae1-3e90d82459dd","Type":"ContainerStarted","Data":"688218f53ae8f18d0dac0d76d103ad7cb4118bbf478272a646c7e70dbce92353"}
Sep 15 11:40:04.214782 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:04.214762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8fbb54f84-trfst" event={"ID":"94211976-fca1-4277-b117-378a811c7800","Type":"ContainerStarted","Data":"a32a9c0820a967cc5c7d8bc0d5b1919651f9e38ee3713d5ea55740200dc62ec5"}
Sep 15 11:40:04.215925 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:04.215902    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c6p6s" event={"ID":"77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2","Type":"ContainerStarted","Data":"619b70ce179b7b87b29f58a24ac8c33616bbf501ac9b30e838c2eb20050615f3"}
Sep 15 11:40:04.231276 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:04.231230    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8d687b44b-k6v8j" podStartSLOduration=1.231216781 podStartE2EDuration="1.231216781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:04.23055335 +0000 UTC m=+93.011124751" watchObservedRunningTime="2026-09-15 11:40:04.231216781 +0000 UTC m=+93.011788166"
Sep 15 11:40:05.226551 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:05.226510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c6p6s" event={"ID":"77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2","Type":"ContainerStarted","Data":"e75ff0f3985b9c588ad0a370d36a6dfaac0fc38de9919a639b645eb788dfbf61"}
Sep 15 11:40:05.244733 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:05.244066    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c6p6s" podStartSLOduration=0.982296683 podStartE2EDuration="2.244045079s" totalImagesPullingTime="1.261748396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:05.24245657 +0000 UTC m=+94.023027958" watchObservedRunningTime="2026-09-15 11:40:05.244045079 +0000 UTC m=+94.024616468"
Sep 15 11:40:07.832778 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:07.832729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:40:07.835307 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:07.835281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e994d99-39e5-4133-9f60-26839d83ab39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7nvb\" (UID: \"2e994d99-39e5-4133-9f60-26839d83ab39\") " pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:40:08.124583 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:08.124507    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7nvb"
Sep 15 11:40:09.123570 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:09.123538    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dvh4t"
Sep 15 11:40:09.475970 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:09.475943    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j7nvb"]
Sep 15 11:40:09.478745 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:40:09.478709    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e994d99_39e5_4133_9f60_26839d83ab39.slice/crio-516c513c42b80b3c56680ffdc77b4dff7f0bda386ae407b8e12836392a6971b0 WatchSource:0}: Error finding container 516c513c42b80b3c56680ffdc77b4dff7f0bda386ae407b8e12836392a6971b0: Status 404 returned error can't find the container with id 516c513c42b80b3c56680ffdc77b4dff7f0bda386ae407b8e12836392a6971b0
Sep 15 11:40:10.052925 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:10.052903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:40:10.055006 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:10.054983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2ea32c9-131c-42c0-b364-7802d07ff5d8-metrics-tls\") pod \"dns-default-6pcgn\" (UID: \"e2ea32c9-131c-42c0-b364-7802d07ff5d8\") " pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:40:10.246542 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:10.246508    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88" event={"ID":"83711006-89e6-4768-bae1-3e90d82459dd","Type":"ContainerStarted","Data":"40330ef1b21e14bbce542ee08e6f6fde4e5a0f5368d77ce722655ef2b3cf6226"}
Sep 15 11:40:10.246991 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:10.246709    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"
Sep 15 11:40:10.248330 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:10.248297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8fbb54f84-trfst" event={"ID":"94211976-fca1-4277-b117-378a811c7800","Type":"ContainerStarted","Data":"2f1f4a8cca837e46e7a44ab08d488deb93c81de30c2c781e4b2417420692d3ec"}
Sep 15 11:40:10.248621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:10.248600    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88"
Sep 15 11:40:10.250084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:10.250032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7nvb" event={"ID":"2e994d99-39e5-4133-9f60-26839d83ab39","Type":"ContainerStarted","Data":"6eef9bf6ead2d3e240012fe0bf9938c2687f895164cd1b0b43e17b3a5a3847d3"}
Sep 15 11:40:10.250084 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:10.250075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7nvb" event={"ID":"2e994d99-39e5-4133-9f60-26839d83ab39","Type":"ContainerStarted","Data":"b79c3eb93f8fd0cf2a8c7742d62bbcac3321654206dc8ef92300318742c3c450"}
Sep 15 11:40:10.250200 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:10.250096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7nvb" event={"ID":"2e994d99-39e5-4133-9f60-26839d83ab39","Type":"ContainerStarted","Data":"516c513c42b80b3c56680ffdc77b4dff7f0bda386ae407b8e12836392a6971b0"}
Sep 15 11:40:10.264774 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:10.264731    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-856c566d88-nlj88" podStartSLOduration=1.692706178 podStartE2EDuration="7.264719344s" totalImagesPullingTime="5.572013166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:10.263561612 +0000 UTC m=+99.044133000" watchObservedRunningTime="2026-09-15 11:40:10.264719344 +0000 UTC m=+99.045290732"
Sep 15 11:40:10.302635 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:10.302582    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8fbb54f84-trfst" podStartSLOduration=1.773100306 podStartE2EDuration="7.30256659s" totalImagesPullingTime="5.529466284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:10.302241188 +0000 UTC m=+99.082812579" watchObservedRunningTime="2026-09-15 11:40:10.30256659 +0000 UTC m=+99.083137978"
Sep 15 11:40:10.332184 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:10.332156    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hx5hb\""
Sep 15 11:40:10.340951 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:10.340931    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:40:10.615621 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:10.615589    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6pcgn"]
Sep 15 11:40:10.618165 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:40:10.618140    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ea32c9_131c_42c0_b364_7802d07ff5d8.slice/crio-7551c0e3766650e0834c5dae8f8cc0b8592b8aff3855858601c7e0271d5049be WatchSource:0}: Error finding container 7551c0e3766650e0834c5dae8f8cc0b8592b8aff3855858601c7e0271d5049be: Status 404 returned error can't find the container with id 7551c0e3766650e0834c5dae8f8cc0b8592b8aff3855858601c7e0271d5049be
Sep 15 11:40:11.255636 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:11.255597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pcgn" event={"ID":"e2ea32c9-131c-42c0-b364-7802d07ff5d8","Type":"ContainerStarted","Data":"7551c0e3766650e0834c5dae8f8cc0b8592b8aff3855858601c7e0271d5049be"}
Sep 15 11:40:12.262153 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:12.262110    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7nvb" event={"ID":"2e994d99-39e5-4133-9f60-26839d83ab39","Type":"ContainerStarted","Data":"613df79636fafeec00b1b9d03f71bd26ee194255746f42cb2abb23240ef68c03"}
Sep 15 11:40:12.281139 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:12.281085    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j7nvb" podStartSLOduration=35.494167531 podStartE2EDuration="37.281068933s" totalImagesPullingTime="1.786901402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:39:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:12.280003714 +0000 UTC m=+101.060575117" watchObservedRunningTime="2026-09-15 11:40:12.281068933 +0000 UTC m=+101.061640321"
Sep 15 11:40:13.066856 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:13.066824    2580 patch_prober.go:28] interesting pod/image-registry-5f86f84bb8-8lcr2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 11:40:13.067045 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:13.066889    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2" podUID="4fe54a34-4168-4be7-ba3d-d8230993b85c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 11:40:13.267692 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:13.267657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pcgn" event={"ID":"e2ea32c9-131c-42c0-b364-7802d07ff5d8","Type":"ContainerStarted","Data":"3765ee8e9c6ba5d79d10d95bf2686df15835e810333602745ddd383d425ded63"}
Sep 15 11:40:13.268130 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:13.267709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pcgn" event={"ID":"e2ea32c9-131c-42c0-b364-7802d07ff5d8","Type":"ContainerStarted","Data":"641ceeb74455140e6c336dba7c5a04b40850cee7b162f101308b59e93d9a847b"}
Sep 15 11:40:13.268130 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:13.267864    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:40:13.286001 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:13.285961    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6pcgn" podStartSLOduration=65.305580921 podStartE2EDuration="1m7.285946537s" totalImagesPullingTime="1.980365616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:13.285321177 +0000 UTC m=+102.065892568" watchObservedRunningTime="2026-09-15 11:40:13.285946537 +0000 UTC m=+102.066517906"
Sep 15 11:40:14.768780 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.768742    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qrj8n"]
Sep 15 11:40:14.772030 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.772003    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.774495 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.774471    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 11:40:14.774822 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.774793    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 11:40:14.774822 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.774790    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bk6sw\""
Sep 15 11:40:14.775022 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.774593    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 11:40:14.775252 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.775233    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 11:40:14.775467 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.775263    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 11:40:14.775467 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.775273    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 11:40:14.891573 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.891547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-wtmp\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.891740 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.891580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ade6c623-622d-4e47-892d-3bb15ccec90b-root\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.891740 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.891614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.891740 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.891664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q6nm\" (UniqueName: \"kubernetes.io/projected/ade6c623-622d-4e47-892d-3bb15ccec90b-kube-api-access-4q6nm\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.891921 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.891804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-tls\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.891921 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.891864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ade6c623-622d-4e47-892d-3bb15ccec90b-sys\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.892025 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.891920    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-accelerators-collector-config\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.892025 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.891973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-textfile\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.892025 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.892015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ade6c623-622d-4e47-892d-3bb15ccec90b-metrics-client-ca\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.992494 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.992462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-wtmp\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.992645 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.992502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ade6c623-622d-4e47-892d-3bb15ccec90b-root\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.992645 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.992524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.992645 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.992547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q6nm\" (UniqueName: \"kubernetes.io/projected/ade6c623-622d-4e47-892d-3bb15ccec90b-kube-api-access-4q6nm\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.992645 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.992622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-tls\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.992881 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.992655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ade6c623-622d-4e47-892d-3bb15ccec90b-sys\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.992881 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.992677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-accelerators-collector-config\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.992881 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.992688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-wtmp\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.992881 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.992752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-textfile\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.992881 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.992815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ade6c623-622d-4e47-892d-3bb15ccec90b-metrics-client-ca\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.993133 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.993036    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ade6c623-622d-4e47-892d-3bb15ccec90b-root\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.993623 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:40:14.993297    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 15 11:40:14.993623 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.993335    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-accelerators-collector-config\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.993623 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:40:14.993362    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-tls podName:ade6c623-622d-4e47-892d-3bb15ccec90b nodeName:}" failed. No retries permitted until 2026-09-15 11:40:15.493342611 +0000 UTC m=+104.273913984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-tls") pod "node-exporter-qrj8n" (UID: "ade6c623-622d-4e47-892d-3bb15ccec90b") : secret "node-exporter-tls" not found
Sep 15 11:40:14.993623 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.993390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ade6c623-622d-4e47-892d-3bb15ccec90b-sys\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.993623 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.993337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-textfile\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.993962 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.993815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ade6c623-622d-4e47-892d-3bb15ccec90b-metrics-client-ca\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:14.995166 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:14.995145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:15.002504 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:15.002482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q6nm\" (UniqueName: \"kubernetes.io/projected/ade6c623-622d-4e47-892d-3bb15ccec90b-kube-api-access-4q6nm\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:15.496381 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:15.496346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-tls\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:15.498669 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:15.498645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ade6c623-622d-4e47-892d-3bb15ccec90b-node-exporter-tls\") pod \"node-exporter-qrj8n\" (UID: \"ade6c623-622d-4e47-892d-3bb15ccec90b\") " pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:15.683167 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:15.683125    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qrj8n"
Sep 15 11:40:15.694363 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:40:15.694321    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade6c623_622d_4e47_892d_3bb15ccec90b.slice/crio-a63f9f42c2323ed73a2c3755a1025185aecaba98d87c60bb2a291e5ceed0dc98 WatchSource:0}: Error finding container a63f9f42c2323ed73a2c3755a1025185aecaba98d87c60bb2a291e5ceed0dc98: Status 404 returned error can't find the container with id a63f9f42c2323ed73a2c3755a1025185aecaba98d87c60bb2a291e5ceed0dc98
Sep 15 11:40:16.278807 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:16.278771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qrj8n" event={"ID":"ade6c623-622d-4e47-892d-3bb15ccec90b","Type":"ContainerStarted","Data":"a63f9f42c2323ed73a2c3755a1025185aecaba98d87c60bb2a291e5ceed0dc98"}
Sep 15 11:40:17.283263 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:17.283229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qrj8n" event={"ID":"ade6c623-622d-4e47-892d-3bb15ccec90b","Type":"ContainerStarted","Data":"154b1d5bd8e8b268c14e7693565077642a4863dd50bd1477dd0176d686f16baa"}
Sep 15 11:40:18.287427 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:18.287389    2580 generic.go:356] "Generic (PLEG): container finished" podID="ade6c623-622d-4e47-892d-3bb15ccec90b" containerID="154b1d5bd8e8b268c14e7693565077642a4863dd50bd1477dd0176d686f16baa" exitCode=0
Sep 15 11:40:18.287791 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:18.287457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qrj8n" event={"ID":"ade6c623-622d-4e47-892d-3bb15ccec90b","Type":"ContainerDied","Data":"154b1d5bd8e8b268c14e7693565077642a4863dd50bd1477dd0176d686f16baa"}
Sep 15 11:40:19.293091 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:19.293053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qrj8n" event={"ID":"ade6c623-622d-4e47-892d-3bb15ccec90b","Type":"ContainerStarted","Data":"d6f3ae0681c191314d59bb68267acc4a1b5ebf9e5de978456a2c0fcf618e3bd5"}
Sep 15 11:40:19.293441 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:19.293119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qrj8n" event={"ID":"ade6c623-622d-4e47-892d-3bb15ccec90b","Type":"ContainerStarted","Data":"4951f39ed83bd2dcedd253041f04ad3b06f08d7871b8f15baddb2e414dd315cf"}
Sep 15 11:40:19.316177 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:19.316131    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qrj8n" podStartSLOduration=3.760521653 podStartE2EDuration="5.316116016s" totalImagesPullingTime="1.493707374s" totalInitContainerRuntime="61.886989ms" isStatefulPod=true podCreationTimestamp="2026-09-15 11:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:19.314377943 +0000 UTC m=+108.094949356" watchObservedRunningTime="2026-09-15 11:40:19.316116016 +0000 UTC m=+108.096687403"
Sep 15 11:40:23.065958 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:23.065930    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:40:23.273652 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:23.273622    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6pcgn"
Sep 15 11:40:24.674398 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.674366    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c55cdf787-wb24p"]
Sep 15 11:40:24.677286 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.677269    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.679432 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.679411    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 11:40:24.679589 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.679569    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 11:40:24.680244 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.680211    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 11:40:24.680417 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.680246    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 11:40:24.680417 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.680269    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 11:40:24.680417 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.680274    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vdw75\""
Sep 15 11:40:24.680603 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.680492    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 11:40:24.680603 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.680535    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 11:40:24.686623 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.686601    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c55cdf787-wb24p"]
Sep 15 11:40:24.771337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.771303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-service-ca\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.771470 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.771373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-serving-cert\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.771470 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.771407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-oauth-config\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.771470 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.771431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjw7r\" (UniqueName: \"kubernetes.io/projected/699030a3-c9ff-4e73-97bd-0995ceffb19f-kube-api-access-gjw7r\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.771580 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.771508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-config\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.771580 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.771531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-oauth-serving-cert\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.872705 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.872667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-serving-cert\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.872807 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.872712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-oauth-config\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.872807 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.872738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjw7r\" (UniqueName: \"kubernetes.io/projected/699030a3-c9ff-4e73-97bd-0995ceffb19f-kube-api-access-gjw7r\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.872807 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.872770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-config\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.872807 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.872792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-oauth-serving-cert\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.873018 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.872847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-service-ca\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.873534 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.873514    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-config\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.873633 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.873515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-service-ca\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.873695 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.873652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-oauth-serving-cert\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.875188 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.875160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-serving-cert\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.875280 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.875215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-oauth-config\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.882638 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.882615    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjw7r\" (UniqueName: \"kubernetes.io/projected/699030a3-c9ff-4e73-97bd-0995ceffb19f-kube-api-access-gjw7r\") pod \"console-6c55cdf787-wb24p\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") " pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:24.986774 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:24.986702    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:25.231259 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:25.231232    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8d687b44b-k6v8j"
Sep 15 11:40:25.255292 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:25.255228    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c55cdf787-wb24p"]
Sep 15 11:40:25.258121 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:40:25.258099    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699030a3_c9ff_4e73_97bd_0995ceffb19f.slice/crio-008bdd797d00bb135ffdc8b7bff0e290b8bc25ef55021697cd41ec46eee096d7 WatchSource:0}: Error finding container 008bdd797d00bb135ffdc8b7bff0e290b8bc25ef55021697cd41ec46eee096d7: Status 404 returned error can't find the container with id 008bdd797d00bb135ffdc8b7bff0e290b8bc25ef55021697cd41ec46eee096d7
Sep 15 11:40:25.314443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:25.314413    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c55cdf787-wb24p" event={"ID":"699030a3-c9ff-4e73-97bd-0995ceffb19f","Type":"ContainerStarted","Data":"008bdd797d00bb135ffdc8b7bff0e290b8bc25ef55021697cd41ec46eee096d7"}
Sep 15 11:40:28.083025 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.082954    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2" podUID="4fe54a34-4168-4be7-ba3d-d8230993b85c" containerName="registry" containerID="cri-o://addd272eaffc02c1f713119b685c2b689c3c1ea1faf381635fa55611d6141f6c" gracePeriod=30
Sep 15 11:40:28.682112 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.682082    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:40:28.702044 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.702017    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxgk8\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-kube-api-access-nxgk8\") pod \"4fe54a34-4168-4be7-ba3d-d8230993b85c\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") "
Sep 15 11:40:28.702166 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.702058    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-certificates\") pod \"4fe54a34-4168-4be7-ba3d-d8230993b85c\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") "
Sep 15 11:40:28.702166 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.702096    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4fe54a34-4168-4be7-ba3d-d8230993b85c-image-registry-private-configuration\") pod \"4fe54a34-4168-4be7-ba3d-d8230993b85c\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") "
Sep 15 11:40:28.702166 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.702124    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls\") pod \"4fe54a34-4168-4be7-ba3d-d8230993b85c\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") "
Sep 15 11:40:28.702166 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.702151    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4fe54a34-4168-4be7-ba3d-d8230993b85c-ca-trust-extracted\") pod \"4fe54a34-4168-4be7-ba3d-d8230993b85c\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") "
Sep 15 11:40:28.702385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.702184    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-bound-sa-token\") pod \"4fe54a34-4168-4be7-ba3d-d8230993b85c\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") "
Sep 15 11:40:28.702385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.702210    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4fe54a34-4168-4be7-ba3d-d8230993b85c-installation-pull-secrets\") pod \"4fe54a34-4168-4be7-ba3d-d8230993b85c\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") "
Sep 15 11:40:28.702385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.702245    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4fe54a34-4168-4be7-ba3d-d8230993b85c-trusted-ca\") pod \"4fe54a34-4168-4be7-ba3d-d8230993b85c\" (UID: \"4fe54a34-4168-4be7-ba3d-d8230993b85c\") "
Sep 15 11:40:28.702530 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.702501    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4fe54a34-4168-4be7-ba3d-d8230993b85c" (UID: "4fe54a34-4168-4be7-ba3d-d8230993b85c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:40:28.702886 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.702818    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe54a34-4168-4be7-ba3d-d8230993b85c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4fe54a34-4168-4be7-ba3d-d8230993b85c" (UID: "4fe54a34-4168-4be7-ba3d-d8230993b85c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:40:28.705760 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.705432    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe54a34-4168-4be7-ba3d-d8230993b85c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4fe54a34-4168-4be7-ba3d-d8230993b85c" (UID: "4fe54a34-4168-4be7-ba3d-d8230993b85c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:40:28.706260 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.706233    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4fe54a34-4168-4be7-ba3d-d8230993b85c" (UID: "4fe54a34-4168-4be7-ba3d-d8230993b85c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:40:28.706661 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.706452    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-kube-api-access-nxgk8" (OuterVolumeSpecName: "kube-api-access-nxgk8") pod "4fe54a34-4168-4be7-ba3d-d8230993b85c" (UID: "4fe54a34-4168-4be7-ba3d-d8230993b85c"). InnerVolumeSpecName "kube-api-access-nxgk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:40:28.706661 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.706521    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4fe54a34-4168-4be7-ba3d-d8230993b85c" (UID: "4fe54a34-4168-4be7-ba3d-d8230993b85c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:40:28.706801 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.706763    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe54a34-4168-4be7-ba3d-d8230993b85c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4fe54a34-4168-4be7-ba3d-d8230993b85c" (UID: "4fe54a34-4168-4be7-ba3d-d8230993b85c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:40:28.714824 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.714801    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe54a34-4168-4be7-ba3d-d8230993b85c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4fe54a34-4168-4be7-ba3d-d8230993b85c" (UID: "4fe54a34-4168-4be7-ba3d-d8230993b85c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:40:28.803281 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.803220    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4fe54a34-4168-4be7-ba3d-d8230993b85c-trusted-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:40:28.803281 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.803242    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxgk8\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-kube-api-access-nxgk8\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:40:28.803281 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.803251    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-certificates\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:40:28.803281 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.803260    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4fe54a34-4168-4be7-ba3d-d8230993b85c-image-registry-private-configuration\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:40:28.803281 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.803272    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-registry-tls\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:40:28.803501 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.803286    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4fe54a34-4168-4be7-ba3d-d8230993b85c-ca-trust-extracted\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:40:28.803501 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.803298    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4fe54a34-4168-4be7-ba3d-d8230993b85c-bound-sa-token\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:40:28.803501 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:28.803308    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4fe54a34-4168-4be7-ba3d-d8230993b85c-installation-pull-secrets\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:40:29.329827 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:29.329788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c55cdf787-wb24p" event={"ID":"699030a3-c9ff-4e73-97bd-0995ceffb19f","Type":"ContainerStarted","Data":"0c0ab3f7d42314ddf12a4c1be9a145b866c7834a6f33a93496e3bccec0d97c6c"}
Sep 15 11:40:29.331083 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:29.331056    2580 generic.go:356] "Generic (PLEG): container finished" podID="4fe54a34-4168-4be7-ba3d-d8230993b85c" containerID="addd272eaffc02c1f713119b685c2b689c3c1ea1faf381635fa55611d6141f6c" exitCode=0
Sep 15 11:40:29.331212 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:29.331098    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2"
Sep 15 11:40:29.331212 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:29.331132    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2" event={"ID":"4fe54a34-4168-4be7-ba3d-d8230993b85c","Type":"ContainerDied","Data":"addd272eaffc02c1f713119b685c2b689c3c1ea1faf381635fa55611d6141f6c"}
Sep 15 11:40:29.331212 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:29.331174    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f86f84bb8-8lcr2" event={"ID":"4fe54a34-4168-4be7-ba3d-d8230993b85c","Type":"ContainerDied","Data":"450e14d49f57d7a4ed0bf9431bb9e162ba5880fa1d3c1c3a8a758ba8b564bc6a"}
Sep 15 11:40:29.331212 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:29.331199    2580 scope.go:117] "RemoveContainer" containerID="addd272eaffc02c1f713119b685c2b689c3c1ea1faf381635fa55611d6141f6c"
Sep 15 11:40:29.344357 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:29.344337    2580 scope.go:117] "RemoveContainer" containerID="addd272eaffc02c1f713119b685c2b689c3c1ea1faf381635fa55611d6141f6c"
Sep 15 11:40:29.344678 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:40:29.344656    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addd272eaffc02c1f713119b685c2b689c3c1ea1faf381635fa55611d6141f6c\": container with ID starting with addd272eaffc02c1f713119b685c2b689c3c1ea1faf381635fa55611d6141f6c not found: ID does not exist" containerID="addd272eaffc02c1f713119b685c2b689c3c1ea1faf381635fa55611d6141f6c"
Sep 15 11:40:29.344742 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:29.344690    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addd272eaffc02c1f713119b685c2b689c3c1ea1faf381635fa55611d6141f6c"} err="failed to get container status \"addd272eaffc02c1f713119b685c2b689c3c1ea1faf381635fa55611d6141f6c\": rpc error: code = NotFound desc = could not find container \"addd272eaffc02c1f713119b685c2b689c3c1ea1faf381635fa55611d6141f6c\": container with ID starting with addd272eaffc02c1f713119b685c2b689c3c1ea1faf381635fa55611d6141f6c not found: ID does not exist"
Sep 15 11:40:29.363926 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:29.363864    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c55cdf787-wb24p" podStartSLOduration=2.02468468 podStartE2EDuration="5.363852871s" totalImagesPullingTime="3.339168191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:29.351690909 +0000 UTC m=+118.132262308" watchObservedRunningTime="2026-09-15 11:40:29.363852871 +0000 UTC m=+118.144424262"
Sep 15 11:40:29.364083 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:29.364070    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f86f84bb8-8lcr2"]
Sep 15 11:40:29.368697 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:29.368673    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f86f84bb8-8lcr2"]
Sep 15 11:40:29.729038 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:29.729009    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fe54a34-4168-4be7-ba3d-d8230993b85c" path="/var/lib/kubelet/pods/4fe54a34-4168-4be7-ba3d-d8230993b85c/volumes"
Sep 15 11:40:34.947016 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:34.946985    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b8ff97898-jdd7m"]
Sep 15 11:40:34.947463 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:34.947242    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe54a34-4168-4be7-ba3d-d8230993b85c" containerName="registry"
Sep 15 11:40:34.947463 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:34.947254    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe54a34-4168-4be7-ba3d-d8230993b85c" containerName="registry"
Sep 15 11:40:34.947463 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:34.947311    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe54a34-4168-4be7-ba3d-d8230993b85c" containerName="registry"
Sep 15 11:40:34.952239 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:34.952219    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:34.959956 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:34.959929    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 15 11:40:34.960371 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:34.960351    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b8ff97898-jdd7m"]
Sep 15 11:40:34.987154 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:34.987129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:34.987264 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:34.987176    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:34.992147 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:34.992127    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:35.048624 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.048595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49e297c3-54df-4118-b513-dc4a741a13bd-console-oauth-config\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.048796 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.048648    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-oauth-serving-cert\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.048796 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.048665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-console-config\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.048796 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.048735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mflfv\" (UniqueName: \"kubernetes.io/projected/49e297c3-54df-4118-b513-dc4a741a13bd-kube-api-access-mflfv\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.048796 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.048767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-trusted-ca-bundle\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.049044 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.048926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-service-ca\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.049044 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.048976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49e297c3-54df-4118-b513-dc4a741a13bd-console-serving-cert\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.149798 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.149774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-console-config\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.149953 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.149816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mflfv\" (UniqueName: \"kubernetes.io/projected/49e297c3-54df-4118-b513-dc4a741a13bd-kube-api-access-mflfv\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.149953 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.149842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-trusted-ca-bundle\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.149953 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.149915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-service-ca\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.149953 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.149948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49e297c3-54df-4118-b513-dc4a741a13bd-console-serving-cert\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.150175 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.149980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49e297c3-54df-4118-b513-dc4a741a13bd-console-oauth-config\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.150175 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.150022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-oauth-serving-cert\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.150583 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.150536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-console-config\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.150693 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.150676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-oauth-serving-cert\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.150742 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.150703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-service-ca\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.150820 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.150801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-trusted-ca-bundle\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.152465 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.152442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49e297c3-54df-4118-b513-dc4a741a13bd-console-serving-cert\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.152551 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.152487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49e297c3-54df-4118-b513-dc4a741a13bd-console-oauth-config\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.158373 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.158354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mflfv\" (UniqueName: \"kubernetes.io/projected/49e297c3-54df-4118-b513-dc4a741a13bd-kube-api-access-mflfv\") pod \"console-7b8ff97898-jdd7m\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") " pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.261016 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.260951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:35.356264 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.356239    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:40:35.530367 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:35.530292    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b8ff97898-jdd7m"]
Sep 15 11:40:35.532918 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:40:35.532891    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e297c3_54df_4118_b513_dc4a741a13bd.slice/crio-2414345a55bb1bd35ff99f879f31fa273b7f958e99f7ebedb409debdcb125a10 WatchSource:0}: Error finding container 2414345a55bb1bd35ff99f879f31fa273b7f958e99f7ebedb409debdcb125a10: Status 404 returned error can't find the container with id 2414345a55bb1bd35ff99f879f31fa273b7f958e99f7ebedb409debdcb125a10
Sep 15 11:40:36.356951 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:36.356915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b8ff97898-jdd7m" event={"ID":"49e297c3-54df-4118-b513-dc4a741a13bd","Type":"ContainerStarted","Data":"94c322dfcc0ffbeebb315a70b2978dcffa6d8345e74ebc5f60fabb4ecfdabcf8"}
Sep 15 11:40:36.357310 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:36.356966    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b8ff97898-jdd7m" event={"ID":"49e297c3-54df-4118-b513-dc4a741a13bd","Type":"ContainerStarted","Data":"2414345a55bb1bd35ff99f879f31fa273b7f958e99f7ebedb409debdcb125a10"}
Sep 15 11:40:36.374945 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:36.374903    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b8ff97898-jdd7m" podStartSLOduration=2.374888561 podStartE2EDuration="2.374888561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:36.374056063 +0000 UTC m=+125.154627448" watchObservedRunningTime="2026-09-15 11:40:36.374888561 +0000 UTC m=+125.155459942"
Sep 15 11:40:41.604499 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:41.604468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs\") pod \"network-metrics-daemon-rxkvf\" (UID: \"1263d93a-e6b5-497d-a022-f24619717fa9\") " pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:40:41.606682 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:41.606661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1263d93a-e6b5-497d-a022-f24619717fa9-metrics-certs\") pod \"network-metrics-daemon-rxkvf\" (UID: \"1263d93a-e6b5-497d-a022-f24619717fa9\") " pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:40:41.838926 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:41.838897    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dmnh2\""
Sep 15 11:40:41.846939 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:41.846920    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxkvf"
Sep 15 11:40:42.181141 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:42.181118    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rxkvf"]
Sep 15 11:40:42.183758 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:40:42.183723    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1263d93a_e6b5_497d_a022_f24619717fa9.slice/crio-4c3e75c0916f795141c79dc48b314faaf0712d82cdcda652f34ca4b5c511debd WatchSource:0}: Error finding container 4c3e75c0916f795141c79dc48b314faaf0712d82cdcda652f34ca4b5c511debd: Status 404 returned error can't find the container with id 4c3e75c0916f795141c79dc48b314faaf0712d82cdcda652f34ca4b5c511debd
Sep 15 11:40:42.378250 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:42.378216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rxkvf" event={"ID":"1263d93a-e6b5-497d-a022-f24619717fa9","Type":"ContainerStarted","Data":"4c3e75c0916f795141c79dc48b314faaf0712d82cdcda652f34ca4b5c511debd"}
Sep 15 11:40:43.383625 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:43.383591    2580 generic.go:356] "Generic (PLEG): container finished" podID="f93710b3-c313-447e-82aa-6a38090ddd60" containerID="3b0daa309511da68f9445def8b0592547c03854e6773d3f44b1cc13124cd0d69" exitCode=0
Sep 15 11:40:43.384007 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:43.383661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8" event={"ID":"f93710b3-c313-447e-82aa-6a38090ddd60","Type":"ContainerDied","Data":"3b0daa309511da68f9445def8b0592547c03854e6773d3f44b1cc13124cd0d69"}
Sep 15 11:40:43.384007 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:43.383996    2580 scope.go:117] "RemoveContainer" containerID="3b0daa309511da68f9445def8b0592547c03854e6773d3f44b1cc13124cd0d69"
Sep 15 11:40:44.388719 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:44.388685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5d8m8" event={"ID":"f93710b3-c313-447e-82aa-6a38090ddd60","Type":"ContainerStarted","Data":"5e2849b34bf7a487e6cc000b60ee8aa944aa322076846d8db45ba2d3ec4f558e"}
Sep 15 11:40:44.390483 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:44.390456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rxkvf" event={"ID":"1263d93a-e6b5-497d-a022-f24619717fa9","Type":"ContainerStarted","Data":"e4471ef96acac2ad16b158e344b7a2374d7ab201c706261f49ca82a0d7fe48e5"}
Sep 15 11:40:44.390690 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:44.390508    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rxkvf" event={"ID":"1263d93a-e6b5-497d-a022-f24619717fa9","Type":"ContainerStarted","Data":"13f225a0292e70b54d5bfd2bedad9a177d02b42a9dc7f52df6974d45ae2bc11b"}
Sep 15 11:40:44.420590 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:44.420532    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rxkvf" podStartSLOduration=131.137475398 podStartE2EDuration="2m12.42051082s" totalImagesPullingTime="1.283035422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:44.419563609 +0000 UTC m=+133.200134997" watchObservedRunningTime="2026-09-15 11:40:44.42051082 +0000 UTC m=+133.201082209"
Sep 15 11:40:45.261238 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:45.261201    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:45.261430 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:45.261255    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:45.266072 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:45.266049    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:45.397513 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:45.397487    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:40:45.448264 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:40:45.448236    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c55cdf787-wb24p"]
Sep 15 11:41:02.456232 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:02.456199    2580 generic.go:356] "Generic (PLEG): container finished" podID="4c462336-f677-41ac-94cb-a3e83fd57a3e" containerID="7c2b71aa4087049315d6647dcbcb6b2e1723428676c6b6f13ec79a92533dc3de" exitCode=0
Sep 15 11:41:02.456636 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:02.456272    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb" event={"ID":"4c462336-f677-41ac-94cb-a3e83fd57a3e","Type":"ContainerDied","Data":"7c2b71aa4087049315d6647dcbcb6b2e1723428676c6b6f13ec79a92533dc3de"}
Sep 15 11:41:02.456636 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:02.456607    2580 scope.go:117] "RemoveContainer" containerID="7c2b71aa4087049315d6647dcbcb6b2e1723428676c6b6f13ec79a92533dc3de"
Sep 15 11:41:03.461347 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:03.461315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-v76nb" event={"ID":"4c462336-f677-41ac-94cb-a3e83fd57a3e","Type":"ContainerStarted","Data":"50b95ca13bfef7593afecc8de4e0026e524911605e34e546ed29b6a3cf5d3e07"}
Sep 15 11:41:10.466899 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.466762    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c55cdf787-wb24p" podUID="699030a3-c9ff-4e73-97bd-0995ceffb19f" containerName="console" containerID="cri-o://0c0ab3f7d42314ddf12a4c1be9a145b866c7834a6f33a93496e3bccec0d97c6c" gracePeriod=15
Sep 15 11:41:10.708965 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.708941    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:41:10.812297 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.812217    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-oauth-config\") pod \"699030a3-c9ff-4e73-97bd-0995ceffb19f\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") "
Sep 15 11:41:10.812297 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.812268    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-oauth-serving-cert\") pod \"699030a3-c9ff-4e73-97bd-0995ceffb19f\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") "
Sep 15 11:41:10.812297 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.812298    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-config\") pod \"699030a3-c9ff-4e73-97bd-0995ceffb19f\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") "
Sep 15 11:41:10.812523 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.812332    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-serving-cert\") pod \"699030a3-c9ff-4e73-97bd-0995ceffb19f\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") "
Sep 15 11:41:10.812523 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.812356    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjw7r\" (UniqueName: \"kubernetes.io/projected/699030a3-c9ff-4e73-97bd-0995ceffb19f-kube-api-access-gjw7r\") pod \"699030a3-c9ff-4e73-97bd-0995ceffb19f\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") "
Sep 15 11:41:10.812523 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.812428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-service-ca\") pod \"699030a3-c9ff-4e73-97bd-0995ceffb19f\" (UID: \"699030a3-c9ff-4e73-97bd-0995ceffb19f\") "
Sep 15 11:41:10.812672 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.812653    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "699030a3-c9ff-4e73-97bd-0995ceffb19f" (UID: "699030a3-c9ff-4e73-97bd-0995ceffb19f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:41:10.812728 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.812689    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-config" (OuterVolumeSpecName: "console-config") pod "699030a3-c9ff-4e73-97bd-0995ceffb19f" (UID: "699030a3-c9ff-4e73-97bd-0995ceffb19f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:41:10.812914 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.812860    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-service-ca" (OuterVolumeSpecName: "service-ca") pod "699030a3-c9ff-4e73-97bd-0995ceffb19f" (UID: "699030a3-c9ff-4e73-97bd-0995ceffb19f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:41:10.814504 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.814477    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "699030a3-c9ff-4e73-97bd-0995ceffb19f" (UID: "699030a3-c9ff-4e73-97bd-0995ceffb19f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:41:10.814591 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.814525    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "699030a3-c9ff-4e73-97bd-0995ceffb19f" (UID: "699030a3-c9ff-4e73-97bd-0995ceffb19f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:41:10.814591 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.814540    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699030a3-c9ff-4e73-97bd-0995ceffb19f-kube-api-access-gjw7r" (OuterVolumeSpecName: "kube-api-access-gjw7r") pod "699030a3-c9ff-4e73-97bd-0995ceffb19f" (UID: "699030a3-c9ff-4e73-97bd-0995ceffb19f"). InnerVolumeSpecName "kube-api-access-gjw7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:41:10.913001 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.912972    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-oauth-config\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:41:10.913001 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.912995    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-oauth-serving-cert\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:41:10.913001 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.913005    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-config\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:41:10.913179 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.913015    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/699030a3-c9ff-4e73-97bd-0995ceffb19f-console-serving-cert\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:41:10.913179 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.913023    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjw7r\" (UniqueName: \"kubernetes.io/projected/699030a3-c9ff-4e73-97bd-0995ceffb19f-kube-api-access-gjw7r\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:41:10.913179 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:10.913031    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/699030a3-c9ff-4e73-97bd-0995ceffb19f-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:41:11.489521 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:11.489489    2580 generic.go:356] "Generic (PLEG): container finished" podID="699030a3-c9ff-4e73-97bd-0995ceffb19f" containerID="0c0ab3f7d42314ddf12a4c1be9a145b866c7834a6f33a93496e3bccec0d97c6c" exitCode=0
Sep 15 11:41:11.489947 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:11.489528    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c55cdf787-wb24p" event={"ID":"699030a3-c9ff-4e73-97bd-0995ceffb19f","Type":"ContainerDied","Data":"0c0ab3f7d42314ddf12a4c1be9a145b866c7834a6f33a93496e3bccec0d97c6c"}
Sep 15 11:41:11.489947 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:11.489534    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c55cdf787-wb24p"
Sep 15 11:41:11.489947 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:11.489569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c55cdf787-wb24p" event={"ID":"699030a3-c9ff-4e73-97bd-0995ceffb19f","Type":"ContainerDied","Data":"008bdd797d00bb135ffdc8b7bff0e290b8bc25ef55021697cd41ec46eee096d7"}
Sep 15 11:41:11.489947 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:11.489590    2580 scope.go:117] "RemoveContainer" containerID="0c0ab3f7d42314ddf12a4c1be9a145b866c7834a6f33a93496e3bccec0d97c6c"
Sep 15 11:41:11.497985 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:11.497968    2580 scope.go:117] "RemoveContainer" containerID="0c0ab3f7d42314ddf12a4c1be9a145b866c7834a6f33a93496e3bccec0d97c6c"
Sep 15 11:41:11.498255 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:41:11.498231    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0ab3f7d42314ddf12a4c1be9a145b866c7834a6f33a93496e3bccec0d97c6c\": container with ID starting with 0c0ab3f7d42314ddf12a4c1be9a145b866c7834a6f33a93496e3bccec0d97c6c not found: ID does not exist" containerID="0c0ab3f7d42314ddf12a4c1be9a145b866c7834a6f33a93496e3bccec0d97c6c"
Sep 15 11:41:11.498316 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:11.498264    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0ab3f7d42314ddf12a4c1be9a145b866c7834a6f33a93496e3bccec0d97c6c"} err="failed to get container status \"0c0ab3f7d42314ddf12a4c1be9a145b866c7834a6f33a93496e3bccec0d97c6c\": rpc error: code = NotFound desc = could not find container \"0c0ab3f7d42314ddf12a4c1be9a145b866c7834a6f33a93496e3bccec0d97c6c\": container with ID starting with 0c0ab3f7d42314ddf12a4c1be9a145b866c7834a6f33a93496e3bccec0d97c6c not found: ID does not exist"
Sep 15 11:41:11.510487 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:11.510467    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c55cdf787-wb24p"]
Sep 15 11:41:11.514865 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:11.514847    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c55cdf787-wb24p"]
Sep 15 11:41:11.728482 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:11.728442    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="699030a3-c9ff-4e73-97bd-0995ceffb19f" path="/var/lib/kubelet/pods/699030a3-c9ff-4e73-97bd-0995ceffb19f/volumes"
Sep 15 11:41:41.919678 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:41.919640    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85f85f7f7f-6z2rm"]
Sep 15 11:41:41.920290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:41.920024    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="699030a3-c9ff-4e73-97bd-0995ceffb19f" containerName="console"
Sep 15 11:41:41.920290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:41.920044    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="699030a3-c9ff-4e73-97bd-0995ceffb19f" containerName="console"
Sep 15 11:41:41.920290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:41.920104    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="699030a3-c9ff-4e73-97bd-0995ceffb19f" containerName="console"
Sep 15 11:41:41.924181 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:41.924161    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:41.932753 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:41.932731    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85f85f7f7f-6z2rm"]
Sep 15 11:41:42.039332 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.039297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f646d17-7968-4db1-8755-444a7b6422b0-console-oauth-config\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.039510 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.039344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f646d17-7968-4db1-8755-444a7b6422b0-console-serving-cert\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.039510 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.039407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-console-config\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.039510 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.039437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-service-ca\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.039510 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.039473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-oauth-serving-cert\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.039510 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.039496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-trusted-ca-bundle\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.039705 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.039519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndcqq\" (UniqueName: \"kubernetes.io/projected/6f646d17-7968-4db1-8755-444a7b6422b0-kube-api-access-ndcqq\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.140555 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.140517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-console-config\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.140555 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.140557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-service-ca\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.140794 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.140581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-oauth-serving-cert\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.140794 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.140598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-trusted-ca-bundle\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.140794 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.140616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndcqq\" (UniqueName: \"kubernetes.io/projected/6f646d17-7968-4db1-8755-444a7b6422b0-kube-api-access-ndcqq\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.140794 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.140646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f646d17-7968-4db1-8755-444a7b6422b0-console-oauth-config\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.140794 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.140667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f646d17-7968-4db1-8755-444a7b6422b0-console-serving-cert\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.141347 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.141316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-service-ca\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.141478 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.141413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-console-config\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.141539 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.141519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-oauth-serving-cert\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.141617 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.141598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-trusted-ca-bundle\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.143185 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.143160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f646d17-7968-4db1-8755-444a7b6422b0-console-oauth-config\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.143326 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.143310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f646d17-7968-4db1-8755-444a7b6422b0-console-serving-cert\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.149930 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.149909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndcqq\" (UniqueName: \"kubernetes.io/projected/6f646d17-7968-4db1-8755-444a7b6422b0-kube-api-access-ndcqq\") pod \"console-85f85f7f7f-6z2rm\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") " pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.233582 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.233500    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:42.505451 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.505410    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85f85f7f7f-6z2rm"]
Sep 15 11:41:42.508052 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:41:42.508025    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f646d17_7968_4db1_8755_444a7b6422b0.slice/crio-f4d72f8cf7c517a4679063fa23cd856dd489bf5d16a4e9e09cb60d8c8e8fabf1 WatchSource:0}: Error finding container f4d72f8cf7c517a4679063fa23cd856dd489bf5d16a4e9e09cb60d8c8e8fabf1: Status 404 returned error can't find the container with id f4d72f8cf7c517a4679063fa23cd856dd489bf5d16a4e9e09cb60d8c8e8fabf1
Sep 15 11:41:42.603224 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.603192    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f85f7f7f-6z2rm" event={"ID":"6f646d17-7968-4db1-8755-444a7b6422b0","Type":"ContainerStarted","Data":"e292358f6a2e715ac0a68178a876efbef26775245dbc4186d8dad3b8d28b59f1"}
Sep 15 11:41:42.603381 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.603243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f85f7f7f-6z2rm" event={"ID":"6f646d17-7968-4db1-8755-444a7b6422b0","Type":"ContainerStarted","Data":"f4d72f8cf7c517a4679063fa23cd856dd489bf5d16a4e9e09cb60d8c8e8fabf1"}
Sep 15 11:41:42.622052 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:42.621995    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85f85f7f7f-6z2rm" podStartSLOduration=1.6219760650000001 podStartE2EDuration="1.621976065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:41:42.620940921 +0000 UTC m=+191.401512311" watchObservedRunningTime="2026-09-15 11:41:42.621976065 +0000 UTC m=+191.402547457"
Sep 15 11:41:52.234429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:52.234393    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:52.234800 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:52.234477    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:52.239249 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:52.239230    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:52.642326 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:52.642250    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:41:52.696383 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:41:52.696349    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b8ff97898-jdd7m"]
Sep 15 11:42:17.717886 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:17.717818    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b8ff97898-jdd7m" podUID="49e297c3-54df-4118-b513-dc4a741a13bd" containerName="console" containerID="cri-o://94c322dfcc0ffbeebb315a70b2978dcffa6d8345e74ebc5f60fabb4ecfdabcf8" gracePeriod=15
Sep 15 11:42:17.964420 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:17.964396    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:42:17.998411 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:17.998358    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-service-ca\") pod \"49e297c3-54df-4118-b513-dc4a741a13bd\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") "
Sep 15 11:42:17.998411 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:17.998397    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49e297c3-54df-4118-b513-dc4a741a13bd-console-oauth-config\") pod \"49e297c3-54df-4118-b513-dc4a741a13bd\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") "
Sep 15 11:42:17.998602 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:17.998418    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-trusted-ca-bundle\") pod \"49e297c3-54df-4118-b513-dc4a741a13bd\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") "
Sep 15 11:42:17.998602 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:17.998442    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-oauth-serving-cert\") pod \"49e297c3-54df-4118-b513-dc4a741a13bd\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") "
Sep 15 11:42:17.998602 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:17.998479    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mflfv\" (UniqueName: \"kubernetes.io/projected/49e297c3-54df-4118-b513-dc4a741a13bd-kube-api-access-mflfv\") pod \"49e297c3-54df-4118-b513-dc4a741a13bd\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") "
Sep 15 11:42:17.998602 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:17.998512    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-console-config\") pod \"49e297c3-54df-4118-b513-dc4a741a13bd\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") "
Sep 15 11:42:17.998602 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:17.998535    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49e297c3-54df-4118-b513-dc4a741a13bd-console-serving-cert\") pod \"49e297c3-54df-4118-b513-dc4a741a13bd\" (UID: \"49e297c3-54df-4118-b513-dc4a741a13bd\") "
Sep 15 11:42:17.999056 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:17.999025    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "49e297c3-54df-4118-b513-dc4a741a13bd" (UID: "49e297c3-54df-4118-b513-dc4a741a13bd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:42:17.999178 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:17.999152    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-console-config" (OuterVolumeSpecName: "console-config") pod "49e297c3-54df-4118-b513-dc4a741a13bd" (UID: "49e297c3-54df-4118-b513-dc4a741a13bd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:42:17.999239 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:17.999204    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "49e297c3-54df-4118-b513-dc4a741a13bd" (UID: "49e297c3-54df-4118-b513-dc4a741a13bd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:42:17.999294 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:17.999237    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-service-ca" (OuterVolumeSpecName: "service-ca") pod "49e297c3-54df-4118-b513-dc4a741a13bd" (UID: "49e297c3-54df-4118-b513-dc4a741a13bd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:42:18.003448 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.003418    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e297c3-54df-4118-b513-dc4a741a13bd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "49e297c3-54df-4118-b513-dc4a741a13bd" (UID: "49e297c3-54df-4118-b513-dc4a741a13bd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:42:18.003552 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.003440    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e297c3-54df-4118-b513-dc4a741a13bd-kube-api-access-mflfv" (OuterVolumeSpecName: "kube-api-access-mflfv") pod "49e297c3-54df-4118-b513-dc4a741a13bd" (UID: "49e297c3-54df-4118-b513-dc4a741a13bd"). InnerVolumeSpecName "kube-api-access-mflfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:42:18.003659 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.003627    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e297c3-54df-4118-b513-dc4a741a13bd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "49e297c3-54df-4118-b513-dc4a741a13bd" (UID: "49e297c3-54df-4118-b513-dc4a741a13bd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:42:18.099120 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.099088    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-trusted-ca-bundle\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:42:18.099120 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.099115    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-oauth-serving-cert\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:42:18.099120 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.099124    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mflfv\" (UniqueName: \"kubernetes.io/projected/49e297c3-54df-4118-b513-dc4a741a13bd-kube-api-access-mflfv\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:42:18.099311 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.099134    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-console-config\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:42:18.099311 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.099143    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49e297c3-54df-4118-b513-dc4a741a13bd-console-serving-cert\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:42:18.099311 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.099152    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49e297c3-54df-4118-b513-dc4a741a13bd-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:42:18.099311 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.099159    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49e297c3-54df-4118-b513-dc4a741a13bd-console-oauth-config\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:42:18.733687 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.733648    2580 generic.go:356] "Generic (PLEG): container finished" podID="49e297c3-54df-4118-b513-dc4a741a13bd" containerID="94c322dfcc0ffbeebb315a70b2978dcffa6d8345e74ebc5f60fabb4ecfdabcf8" exitCode=0
Sep 15 11:42:18.734078 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.733696    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b8ff97898-jdd7m"
Sep 15 11:42:18.734078 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.733715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b8ff97898-jdd7m" event={"ID":"49e297c3-54df-4118-b513-dc4a741a13bd","Type":"ContainerDied","Data":"94c322dfcc0ffbeebb315a70b2978dcffa6d8345e74ebc5f60fabb4ecfdabcf8"}
Sep 15 11:42:18.734078 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.733760    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b8ff97898-jdd7m" event={"ID":"49e297c3-54df-4118-b513-dc4a741a13bd","Type":"ContainerDied","Data":"2414345a55bb1bd35ff99f879f31fa273b7f958e99f7ebedb409debdcb125a10"}
Sep 15 11:42:18.734078 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.733786    2580 scope.go:117] "RemoveContainer" containerID="94c322dfcc0ffbeebb315a70b2978dcffa6d8345e74ebc5f60fabb4ecfdabcf8"
Sep 15 11:42:18.743535 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.743513    2580 scope.go:117] "RemoveContainer" containerID="94c322dfcc0ffbeebb315a70b2978dcffa6d8345e74ebc5f60fabb4ecfdabcf8"
Sep 15 11:42:18.743794 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:42:18.743777    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c322dfcc0ffbeebb315a70b2978dcffa6d8345e74ebc5f60fabb4ecfdabcf8\": container with ID starting with 94c322dfcc0ffbeebb315a70b2978dcffa6d8345e74ebc5f60fabb4ecfdabcf8 not found: ID does not exist" containerID="94c322dfcc0ffbeebb315a70b2978dcffa6d8345e74ebc5f60fabb4ecfdabcf8"
Sep 15 11:42:18.743868 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.743801    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c322dfcc0ffbeebb315a70b2978dcffa6d8345e74ebc5f60fabb4ecfdabcf8"} err="failed to get container status \"94c322dfcc0ffbeebb315a70b2978dcffa6d8345e74ebc5f60fabb4ecfdabcf8\": rpc error: code = NotFound desc = could not find container \"94c322dfcc0ffbeebb315a70b2978dcffa6d8345e74ebc5f60fabb4ecfdabcf8\": container with ID starting with 94c322dfcc0ffbeebb315a70b2978dcffa6d8345e74ebc5f60fabb4ecfdabcf8 not found: ID does not exist"
Sep 15 11:42:18.757734 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.757687    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b8ff97898-jdd7m"]
Sep 15 11:42:18.763271 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:18.761292    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b8ff97898-jdd7m"]
Sep 15 11:42:19.729602 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:42:19.729567    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49e297c3-54df-4118-b513-dc4a741a13bd" path="/var/lib/kubelet/pods/49e297c3-54df-4118-b513-dc4a741a13bd/volumes"
Sep 15 11:43:18.048078 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.048042    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c58cb7d7d-wx8f2"]
Sep 15 11:43:18.048497 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.048302    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49e297c3-54df-4118-b513-dc4a741a13bd" containerName="console"
Sep 15 11:43:18.048497 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.048313    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="49e297c3-54df-4118-b513-dc4a741a13bd" containerName="console"
Sep 15 11:43:18.048497 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.048355    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="49e297c3-54df-4118-b513-dc4a741a13bd" containerName="console"
Sep 15 11:43:18.051097 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.051079    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.062194 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.062171    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c58cb7d7d-wx8f2"]
Sep 15 11:43:18.211963 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.211927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-service-ca\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.212156 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.211970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/946d2972-ab99-4f44-a10a-16c418cfd80c-console-serving-cert\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.212156 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.211992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/946d2972-ab99-4f44-a10a-16c418cfd80c-console-oauth-config\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.212156 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.212010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-trusted-ca-bundle\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.212156 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.212084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-oauth-serving-cert\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.212156 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.212128    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd9gc\" (UniqueName: \"kubernetes.io/projected/946d2972-ab99-4f44-a10a-16c418cfd80c-kube-api-access-rd9gc\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.212354 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.212204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-console-config\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.313074 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.312983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-oauth-serving-cert\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.313074 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.313024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd9gc\" (UniqueName: \"kubernetes.io/projected/946d2972-ab99-4f44-a10a-16c418cfd80c-kube-api-access-rd9gc\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.313074 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.313064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-console-config\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.313346 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.313105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-service-ca\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.313346 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.313144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/946d2972-ab99-4f44-a10a-16c418cfd80c-console-serving-cert\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.313346 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.313171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/946d2972-ab99-4f44-a10a-16c418cfd80c-console-oauth-config\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.313346 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.313199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-trusted-ca-bundle\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.313782 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.313759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-oauth-serving-cert\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.313782 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.313771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-console-config\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.313951 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.313853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-service-ca\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.314048 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.314028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-trusted-ca-bundle\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.315596 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.315575    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/946d2972-ab99-4f44-a10a-16c418cfd80c-console-oauth-config\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.315739 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.315723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/946d2972-ab99-4f44-a10a-16c418cfd80c-console-serving-cert\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.321900 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.321859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd9gc\" (UniqueName: \"kubernetes.io/projected/946d2972-ab99-4f44-a10a-16c418cfd80c-kube-api-access-rd9gc\") pod \"console-6c58cb7d7d-wx8f2\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") " pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.359457 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.359436    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:18.628602 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.628530    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c58cb7d7d-wx8f2"]
Sep 15 11:43:18.631632 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:43:18.631605    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946d2972_ab99_4f44_a10a_16c418cfd80c.slice/crio-3e34d5b68966cc367fd94b38a23bb76faef16151d55b572cf680589c7ec218c7 WatchSource:0}: Error finding container 3e34d5b68966cc367fd94b38a23bb76faef16151d55b572cf680589c7ec218c7: Status 404 returned error can't find the container with id 3e34d5b68966cc367fd94b38a23bb76faef16151d55b572cf680589c7ec218c7
Sep 15 11:43:18.933987 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.933948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c58cb7d7d-wx8f2" event={"ID":"946d2972-ab99-4f44-a10a-16c418cfd80c","Type":"ContainerStarted","Data":"624e08e1ee5b6029004ea61ca37b5c7cab63364df669a9a226c347992604b9e9"}
Sep 15 11:43:18.934162 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.934004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c58cb7d7d-wx8f2" event={"ID":"946d2972-ab99-4f44-a10a-16c418cfd80c","Type":"ContainerStarted","Data":"3e34d5b68966cc367fd94b38a23bb76faef16151d55b572cf680589c7ec218c7"}
Sep 15 11:43:18.956646 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:18.956598    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c58cb7d7d-wx8f2" podStartSLOduration=0.956583776 podStartE2EDuration="956.583776ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:43:18.954758111 +0000 UTC m=+287.735329499" watchObservedRunningTime="2026-09-15 11:43:18.956583776 +0000 UTC m=+287.737155164"
Sep 15 11:43:28.360390 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:28.360353    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:28.360767 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:28.360417    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:28.365167 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:28.365146    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:28.976215 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:28.976171    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:43:29.025822 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:29.025782    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85f85f7f7f-6z2rm"]
Sep 15 11:43:31.613534 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:31.613502    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 11:43:54.044239 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.044175    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85f85f7f7f-6z2rm" podUID="6f646d17-7968-4db1-8755-444a7b6422b0" containerName="console" containerID="cri-o://e292358f6a2e715ac0a68178a876efbef26775245dbc4186d8dad3b8d28b59f1" gracePeriod=15
Sep 15 11:43:54.275010 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.274988    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:43:54.368228 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.368154    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f646d17-7968-4db1-8755-444a7b6422b0-console-serving-cert\") pod \"6f646d17-7968-4db1-8755-444a7b6422b0\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") "
Sep 15 11:43:54.368228 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.368206    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f646d17-7968-4db1-8755-444a7b6422b0-console-oauth-config\") pod \"6f646d17-7968-4db1-8755-444a7b6422b0\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") "
Sep 15 11:43:54.368423 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.368231    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-console-config\") pod \"6f646d17-7968-4db1-8755-444a7b6422b0\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") "
Sep 15 11:43:54.368423 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.368270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndcqq\" (UniqueName: \"kubernetes.io/projected/6f646d17-7968-4db1-8755-444a7b6422b0-kube-api-access-ndcqq\") pod \"6f646d17-7968-4db1-8755-444a7b6422b0\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") "
Sep 15 11:43:54.368637 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.368609    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-console-config" (OuterVolumeSpecName: "console-config") pod "6f646d17-7968-4db1-8755-444a7b6422b0" (UID: "6f646d17-7968-4db1-8755-444a7b6422b0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:43:54.370292 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.370258    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f646d17-7968-4db1-8755-444a7b6422b0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6f646d17-7968-4db1-8755-444a7b6422b0" (UID: "6f646d17-7968-4db1-8755-444a7b6422b0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:43:54.370383 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.370316    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f646d17-7968-4db1-8755-444a7b6422b0-kube-api-access-ndcqq" (OuterVolumeSpecName: "kube-api-access-ndcqq") pod "6f646d17-7968-4db1-8755-444a7b6422b0" (UID: "6f646d17-7968-4db1-8755-444a7b6422b0"). InnerVolumeSpecName "kube-api-access-ndcqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:43:54.370383 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.370318    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f646d17-7968-4db1-8755-444a7b6422b0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6f646d17-7968-4db1-8755-444a7b6422b0" (UID: "6f646d17-7968-4db1-8755-444a7b6422b0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:43:54.469418 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.469394    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-oauth-serving-cert\") pod \"6f646d17-7968-4db1-8755-444a7b6422b0\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") "
Sep 15 11:43:54.469534 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.469426    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-trusted-ca-bundle\") pod \"6f646d17-7968-4db1-8755-444a7b6422b0\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") "
Sep 15 11:43:54.469534 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.469454    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-service-ca\") pod \"6f646d17-7968-4db1-8755-444a7b6422b0\" (UID: \"6f646d17-7968-4db1-8755-444a7b6422b0\") "
Sep 15 11:43:54.469675 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.469660    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f646d17-7968-4db1-8755-444a7b6422b0-console-serving-cert\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:43:54.469731 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.469681    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f646d17-7968-4db1-8755-444a7b6422b0-console-oauth-config\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:43:54.469731 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.469694    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-console-config\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:43:54.469731 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.469708    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndcqq\" (UniqueName: \"kubernetes.io/projected/6f646d17-7968-4db1-8755-444a7b6422b0-kube-api-access-ndcqq\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:43:54.469731 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.469706    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6f646d17-7968-4db1-8755-444a7b6422b0" (UID: "6f646d17-7968-4db1-8755-444a7b6422b0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:43:54.469731 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.469722    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6f646d17-7968-4db1-8755-444a7b6422b0" (UID: "6f646d17-7968-4db1-8755-444a7b6422b0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:43:54.469906 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.469762    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-service-ca" (OuterVolumeSpecName: "service-ca") pod "6f646d17-7968-4db1-8755-444a7b6422b0" (UID: "6f646d17-7968-4db1-8755-444a7b6422b0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:43:54.570207 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.570179    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-oauth-serving-cert\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:43:54.570207 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.570203    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-trusted-ca-bundle\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:43:54.570360 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:54.570216    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f646d17-7968-4db1-8755-444a7b6422b0-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:43:55.065230 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:55.065195    2580 generic.go:356] "Generic (PLEG): container finished" podID="6f646d17-7968-4db1-8755-444a7b6422b0" containerID="e292358f6a2e715ac0a68178a876efbef26775245dbc4186d8dad3b8d28b59f1" exitCode=0
Sep 15 11:43:55.065699 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:55.065257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f85f7f7f-6z2rm" event={"ID":"6f646d17-7968-4db1-8755-444a7b6422b0","Type":"ContainerDied","Data":"e292358f6a2e715ac0a68178a876efbef26775245dbc4186d8dad3b8d28b59f1"}
Sep 15 11:43:55.065699 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:55.065267    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f85f7f7f-6z2rm"
Sep 15 11:43:55.065699 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:55.065295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f85f7f7f-6z2rm" event={"ID":"6f646d17-7968-4db1-8755-444a7b6422b0","Type":"ContainerDied","Data":"f4d72f8cf7c517a4679063fa23cd856dd489bf5d16a4e9e09cb60d8c8e8fabf1"}
Sep 15 11:43:55.065699 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:55.065314    2580 scope.go:117] "RemoveContainer" containerID="e292358f6a2e715ac0a68178a876efbef26775245dbc4186d8dad3b8d28b59f1"
Sep 15 11:43:55.073358 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:55.073341    2580 scope.go:117] "RemoveContainer" containerID="e292358f6a2e715ac0a68178a876efbef26775245dbc4186d8dad3b8d28b59f1"
Sep 15 11:43:55.073615 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:43:55.073596    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e292358f6a2e715ac0a68178a876efbef26775245dbc4186d8dad3b8d28b59f1\": container with ID starting with e292358f6a2e715ac0a68178a876efbef26775245dbc4186d8dad3b8d28b59f1 not found: ID does not exist" containerID="e292358f6a2e715ac0a68178a876efbef26775245dbc4186d8dad3b8d28b59f1"
Sep 15 11:43:55.073678 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:55.073622    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e292358f6a2e715ac0a68178a876efbef26775245dbc4186d8dad3b8d28b59f1"} err="failed to get container status \"e292358f6a2e715ac0a68178a876efbef26775245dbc4186d8dad3b8d28b59f1\": rpc error: code = NotFound desc = could not find container \"e292358f6a2e715ac0a68178a876efbef26775245dbc4186d8dad3b8d28b59f1\": container with ID starting with e292358f6a2e715ac0a68178a876efbef26775245dbc4186d8dad3b8d28b59f1 not found: ID does not exist"
Sep 15 11:43:55.086476 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:55.086454    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85f85f7f7f-6z2rm"]
Sep 15 11:43:55.092257 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:55.092235    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85f85f7f7f-6z2rm"]
Sep 15 11:43:55.728792 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:55.728763    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f646d17-7968-4db1-8755-444a7b6422b0" path="/var/lib/kubelet/pods/6f646d17-7968-4db1-8755-444a7b6422b0/volumes"
Sep 15 11:43:57.991282 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:57.991250    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"]
Sep 15 11:43:57.991636 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:57.991492    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f646d17-7968-4db1-8755-444a7b6422b0" containerName="console"
Sep 15 11:43:57.991636 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:57.991502    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f646d17-7968-4db1-8755-444a7b6422b0" containerName="console"
Sep 15 11:43:57.991636 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:57.991554    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f646d17-7968-4db1-8755-444a7b6422b0" containerName="console"
Sep 15 11:43:57.995415 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:57.995396    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"
Sep 15 11:43:57.997809 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:57.997782    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 11:43:57.998514 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:57.998495    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bzsrv\""
Sep 15 11:43:57.998622 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:57.998535    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 11:43:58.005923 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:58.005901    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"]
Sep 15 11:43:58.096040 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:58.096010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/057ba945-3a7f-4db7-a3b4-e152650cdafc-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2\" (UID: \"057ba945-3a7f-4db7-a3b4-e152650cdafc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"
Sep 15 11:43:58.096040 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:58.096042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nj55\" (UniqueName: \"kubernetes.io/projected/057ba945-3a7f-4db7-a3b4-e152650cdafc-kube-api-access-6nj55\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2\" (UID: \"057ba945-3a7f-4db7-a3b4-e152650cdafc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"
Sep 15 11:43:58.096234 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:58.096089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/057ba945-3a7f-4db7-a3b4-e152650cdafc-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2\" (UID: \"057ba945-3a7f-4db7-a3b4-e152650cdafc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"
Sep 15 11:43:58.197308 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:58.197281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/057ba945-3a7f-4db7-a3b4-e152650cdafc-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2\" (UID: \"057ba945-3a7f-4db7-a3b4-e152650cdafc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"
Sep 15 11:43:58.197416 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:58.197328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/057ba945-3a7f-4db7-a3b4-e152650cdafc-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2\" (UID: \"057ba945-3a7f-4db7-a3b4-e152650cdafc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"
Sep 15 11:43:58.197416 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:58.197347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nj55\" (UniqueName: \"kubernetes.io/projected/057ba945-3a7f-4db7-a3b4-e152650cdafc-kube-api-access-6nj55\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2\" (UID: \"057ba945-3a7f-4db7-a3b4-e152650cdafc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"
Sep 15 11:43:58.197735 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:58.197713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/057ba945-3a7f-4db7-a3b4-e152650cdafc-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2\" (UID: \"057ba945-3a7f-4db7-a3b4-e152650cdafc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"
Sep 15 11:43:58.197816 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:58.197723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/057ba945-3a7f-4db7-a3b4-e152650cdafc-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2\" (UID: \"057ba945-3a7f-4db7-a3b4-e152650cdafc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"
Sep 15 11:43:58.207377 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:58.207349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nj55\" (UniqueName: \"kubernetes.io/projected/057ba945-3a7f-4db7-a3b4-e152650cdafc-kube-api-access-6nj55\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2\" (UID: \"057ba945-3a7f-4db7-a3b4-e152650cdafc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"
Sep 15 11:43:58.304191 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:58.304128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"
Sep 15 11:43:58.577336 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:58.577253    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"]
Sep 15 11:43:58.580227 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:43:58.580199    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057ba945_3a7f_4db7_a3b4_e152650cdafc.slice/crio-8b38c7b99c61427a5c7514fd67f34a67253d02661a81347f62608a2bfd70ebe6 WatchSource:0}: Error finding container 8b38c7b99c61427a5c7514fd67f34a67253d02661a81347f62608a2bfd70ebe6: Status 404 returned error can't find the container with id 8b38c7b99c61427a5c7514fd67f34a67253d02661a81347f62608a2bfd70ebe6
Sep 15 11:43:58.582639 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:58.582618    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 11:43:59.082259 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:43:59.082216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2" event={"ID":"057ba945-3a7f-4db7-a3b4-e152650cdafc","Type":"ContainerStarted","Data":"8b38c7b99c61427a5c7514fd67f34a67253d02661a81347f62608a2bfd70ebe6"}
Sep 15 11:44:05.111648 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:05.111607    2580 generic.go:356] "Generic (PLEG): container finished" podID="057ba945-3a7f-4db7-a3b4-e152650cdafc" containerID="cdc94bb16e8902dc01779ceba3dba17dc49786ead33a64da1b442c4c72141082" exitCode=0
Sep 15 11:44:05.112162 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:05.111665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2" event={"ID":"057ba945-3a7f-4db7-a3b4-e152650cdafc","Type":"ContainerDied","Data":"cdc94bb16e8902dc01779ceba3dba17dc49786ead33a64da1b442c4c72141082"}
Sep 15 11:44:08.126064 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:08.126027    2580 generic.go:356] "Generic (PLEG): container finished" podID="057ba945-3a7f-4db7-a3b4-e152650cdafc" containerID="c28f0da00af675a096e66b73716fabeeab4d10a5edcb2b6bf72b11e38fd3f14c" exitCode=0
Sep 15 11:44:08.126488 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:08.126106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2" event={"ID":"057ba945-3a7f-4db7-a3b4-e152650cdafc","Type":"ContainerDied","Data":"c28f0da00af675a096e66b73716fabeeab4d10a5edcb2b6bf72b11e38fd3f14c"}
Sep 15 11:44:15.159205 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:15.159124    2580 generic.go:356] "Generic (PLEG): container finished" podID="057ba945-3a7f-4db7-a3b4-e152650cdafc" containerID="46bdf86cca0740d1477ef3ca53149bd3d85770cbc03d8ba3e1b083ca5b895659" exitCode=0
Sep 15 11:44:15.159205 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:15.159164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2" event={"ID":"057ba945-3a7f-4db7-a3b4-e152650cdafc","Type":"ContainerDied","Data":"46bdf86cca0740d1477ef3ca53149bd3d85770cbc03d8ba3e1b083ca5b895659"}
Sep 15 11:44:16.317368 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:16.317335    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"
Sep 15 11:44:16.441793 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:16.441706    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nj55\" (UniqueName: \"kubernetes.io/projected/057ba945-3a7f-4db7-a3b4-e152650cdafc-kube-api-access-6nj55\") pod \"057ba945-3a7f-4db7-a3b4-e152650cdafc\" (UID: \"057ba945-3a7f-4db7-a3b4-e152650cdafc\") "
Sep 15 11:44:16.441978 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:16.441807    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/057ba945-3a7f-4db7-a3b4-e152650cdafc-bundle\") pod \"057ba945-3a7f-4db7-a3b4-e152650cdafc\" (UID: \"057ba945-3a7f-4db7-a3b4-e152650cdafc\") "
Sep 15 11:44:16.441978 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:16.441846    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/057ba945-3a7f-4db7-a3b4-e152650cdafc-util\") pod \"057ba945-3a7f-4db7-a3b4-e152650cdafc\" (UID: \"057ba945-3a7f-4db7-a3b4-e152650cdafc\") "
Sep 15 11:44:16.442452 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:16.442414    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057ba945-3a7f-4db7-a3b4-e152650cdafc-bundle" (OuterVolumeSpecName: "bundle") pod "057ba945-3a7f-4db7-a3b4-e152650cdafc" (UID: "057ba945-3a7f-4db7-a3b4-e152650cdafc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:44:16.443951 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:16.443923    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057ba945-3a7f-4db7-a3b4-e152650cdafc-kube-api-access-6nj55" (OuterVolumeSpecName: "kube-api-access-6nj55") pod "057ba945-3a7f-4db7-a3b4-e152650cdafc" (UID: "057ba945-3a7f-4db7-a3b4-e152650cdafc"). InnerVolumeSpecName "kube-api-access-6nj55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:44:16.447257 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:16.447220    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057ba945-3a7f-4db7-a3b4-e152650cdafc-util" (OuterVolumeSpecName: "util") pod "057ba945-3a7f-4db7-a3b4-e152650cdafc" (UID: "057ba945-3a7f-4db7-a3b4-e152650cdafc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:44:16.542851 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:16.542818    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/057ba945-3a7f-4db7-a3b4-e152650cdafc-bundle\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:44:16.542851 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:16.542848    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/057ba945-3a7f-4db7-a3b4-e152650cdafc-util\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:44:16.542851 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:16.542857    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nj55\" (UniqueName: \"kubernetes.io/projected/057ba945-3a7f-4db7-a3b4-e152650cdafc-kube-api-access-6nj55\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:44:17.168401 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:17.168375    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2"
Sep 15 11:44:17.168565 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:17.168380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402phc2" event={"ID":"057ba945-3a7f-4db7-a3b4-e152650cdafc","Type":"ContainerDied","Data":"8b38c7b99c61427a5c7514fd67f34a67253d02661a81347f62608a2bfd70ebe6"}
Sep 15 11:44:17.168565 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:17.168490    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b38c7b99c61427a5c7514fd67f34a67253d02661a81347f62608a2bfd70ebe6"
Sep 15 11:44:21.047544 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.047514    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wp45n"]
Sep 15 11:44:21.048007 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.047744    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="057ba945-3a7f-4db7-a3b4-e152650cdafc" containerName="util"
Sep 15 11:44:21.048007 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.047754    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="057ba945-3a7f-4db7-a3b4-e152650cdafc" containerName="util"
Sep 15 11:44:21.048007 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.047768    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="057ba945-3a7f-4db7-a3b4-e152650cdafc" containerName="extract"
Sep 15 11:44:21.048007 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.047774    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="057ba945-3a7f-4db7-a3b4-e152650cdafc" containerName="extract"
Sep 15 11:44:21.048007 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.047788    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="057ba945-3a7f-4db7-a3b4-e152650cdafc" containerName="pull"
Sep 15 11:44:21.048007 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.047794    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="057ba945-3a7f-4db7-a3b4-e152650cdafc" containerName="pull"
Sep 15 11:44:21.048007 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.047832    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="057ba945-3a7f-4db7-a3b4-e152650cdafc" containerName="extract"
Sep 15 11:44:21.050237 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.050221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wp45n"
Sep 15 11:44:21.052579 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.052550    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 15 11:44:21.052744 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.052586    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:44:21.052817 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.052808    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-48vsn\""
Sep 15 11:44:21.064027 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.064003    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wp45n"]
Sep 15 11:44:21.176335 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.176302    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3788897-c577-496c-ba07-1b8416fb1c6c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wp45n\" (UID: \"c3788897-c577-496c-ba07-1b8416fb1c6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wp45n"
Sep 15 11:44:21.176486 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.176342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcqqk\" (UniqueName: \"kubernetes.io/projected/c3788897-c577-496c-ba07-1b8416fb1c6c-kube-api-access-qcqqk\") pod \"cert-manager-operator-controller-manager-55c444bf86-wp45n\" (UID: \"c3788897-c577-496c-ba07-1b8416fb1c6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wp45n"
Sep 15 11:44:21.277395 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.277368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcqqk\" (UniqueName: \"kubernetes.io/projected/c3788897-c577-496c-ba07-1b8416fb1c6c-kube-api-access-qcqqk\") pod \"cert-manager-operator-controller-manager-55c444bf86-wp45n\" (UID: \"c3788897-c577-496c-ba07-1b8416fb1c6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wp45n"
Sep 15 11:44:21.277534 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.277422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3788897-c577-496c-ba07-1b8416fb1c6c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wp45n\" (UID: \"c3788897-c577-496c-ba07-1b8416fb1c6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wp45n"
Sep 15 11:44:21.277725 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.277710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3788897-c577-496c-ba07-1b8416fb1c6c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wp45n\" (UID: \"c3788897-c577-496c-ba07-1b8416fb1c6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wp45n"
Sep 15 11:44:21.285699 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.285667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcqqk\" (UniqueName: \"kubernetes.io/projected/c3788897-c577-496c-ba07-1b8416fb1c6c-kube-api-access-qcqqk\") pod \"cert-manager-operator-controller-manager-55c444bf86-wp45n\" (UID: \"c3788897-c577-496c-ba07-1b8416fb1c6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wp45n"
Sep 15 11:44:21.359171 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.359115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wp45n"
Sep 15 11:44:21.630372 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:21.630307    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wp45n"]
Sep 15 11:44:21.634062 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:44:21.634034    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3788897_c577_496c_ba07_1b8416fb1c6c.slice/crio-da7b4ef3aa6dbedbf554616a670a1545c0f5135a26c8e1c9b8492783ae87e301 WatchSource:0}: Error finding container da7b4ef3aa6dbedbf554616a670a1545c0f5135a26c8e1c9b8492783ae87e301: Status 404 returned error can't find the container with id da7b4ef3aa6dbedbf554616a670a1545c0f5135a26c8e1c9b8492783ae87e301
Sep 15 11:44:22.188246 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:22.188204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wp45n" event={"ID":"c3788897-c577-496c-ba07-1b8416fb1c6c","Type":"ContainerStarted","Data":"da7b4ef3aa6dbedbf554616a670a1545c0f5135a26c8e1c9b8492783ae87e301"}
Sep 15 11:44:24.199890 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:24.199785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wp45n" event={"ID":"c3788897-c577-496c-ba07-1b8416fb1c6c","Type":"ContainerStarted","Data":"c3378ca6c6e09677f5034e91bed6df443aaaf15d4f15c5b9e4d02260ea121117"}
Sep 15 11:44:24.225861 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:24.225803    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wp45n" podStartSLOduration=1.482415953 podStartE2EDuration="3.22578646s" totalImagesPullingTime="1.743370507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:44:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:44:24.223229888 +0000 UTC m=+353.003801276" watchObservedRunningTime="2026-09-15 11:44:24.22578646 +0000 UTC m=+353.006357848"
Sep 15 11:44:25.320204 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.320171    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"]
Sep 15 11:44:25.322341 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.322325    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"
Sep 15 11:44:25.324538 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.324514    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 11:44:25.325138 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.325121    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 11:44:25.325224 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.325120    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bzsrv\""
Sep 15 11:44:25.333994 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.333974    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"]
Sep 15 11:44:25.405583 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.405557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff8be1d6-0235-434a-8c16-a35a3769423a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm\" (UID: \"ff8be1d6-0235-434a-8c16-a35a3769423a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"
Sep 15 11:44:25.405719 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.405596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff8be1d6-0235-434a-8c16-a35a3769423a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm\" (UID: \"ff8be1d6-0235-434a-8c16-a35a3769423a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"
Sep 15 11:44:25.405719 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.405638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr44w\" (UniqueName: \"kubernetes.io/projected/ff8be1d6-0235-434a-8c16-a35a3769423a-kube-api-access-qr44w\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm\" (UID: \"ff8be1d6-0235-434a-8c16-a35a3769423a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"
Sep 15 11:44:25.506825 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.506786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff8be1d6-0235-434a-8c16-a35a3769423a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm\" (UID: \"ff8be1d6-0235-434a-8c16-a35a3769423a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"
Sep 15 11:44:25.507007 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.506845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr44w\" (UniqueName: \"kubernetes.io/projected/ff8be1d6-0235-434a-8c16-a35a3769423a-kube-api-access-qr44w\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm\" (UID: \"ff8be1d6-0235-434a-8c16-a35a3769423a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"
Sep 15 11:44:25.507007 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.506903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff8be1d6-0235-434a-8c16-a35a3769423a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm\" (UID: \"ff8be1d6-0235-434a-8c16-a35a3769423a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"
Sep 15 11:44:25.507263 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.507248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff8be1d6-0235-434a-8c16-a35a3769423a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm\" (UID: \"ff8be1d6-0235-434a-8c16-a35a3769423a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"
Sep 15 11:44:25.507303 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.507262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff8be1d6-0235-434a-8c16-a35a3769423a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm\" (UID: \"ff8be1d6-0235-434a-8c16-a35a3769423a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"
Sep 15 11:44:25.515511 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.515485    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr44w\" (UniqueName: \"kubernetes.io/projected/ff8be1d6-0235-434a-8c16-a35a3769423a-kube-api-access-qr44w\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm\" (UID: \"ff8be1d6-0235-434a-8c16-a35a3769423a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"
Sep 15 11:44:25.630470 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.630412    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"
Sep 15 11:44:25.897191 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:25.897112    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"]
Sep 15 11:44:25.900644 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:44:25.900618    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8be1d6_0235_434a_8c16_a35a3769423a.slice/crio-6a0732541c1114a1333c344e31e3ebf19fb91659bef6292cf323f437957b9a4f WatchSource:0}: Error finding container 6a0732541c1114a1333c344e31e3ebf19fb91659bef6292cf323f437957b9a4f: Status 404 returned error can't find the container with id 6a0732541c1114a1333c344e31e3ebf19fb91659bef6292cf323f437957b9a4f
Sep 15 11:44:26.209657 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:26.209627    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff8be1d6-0235-434a-8c16-a35a3769423a" containerID="0102dfef50e4769dba9ad0de398c222bff7c23443bd33e07529bf4877dd8c1e5" exitCode=0
Sep 15 11:44:26.209833 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:26.209693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm" event={"ID":"ff8be1d6-0235-434a-8c16-a35a3769423a","Type":"ContainerDied","Data":"0102dfef50e4769dba9ad0de398c222bff7c23443bd33e07529bf4877dd8c1e5"}
Sep 15 11:44:26.209833 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:26.209733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm" event={"ID":"ff8be1d6-0235-434a-8c16-a35a3769423a","Type":"ContainerStarted","Data":"6a0732541c1114a1333c344e31e3ebf19fb91659bef6292cf323f437957b9a4f"}
Sep 15 11:44:28.219323 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:28.219288    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff8be1d6-0235-434a-8c16-a35a3769423a" containerID="30f0d05f5f6c9fa81c388fdc466011a0093a9e9e4bde15510b95dd578fff468a" exitCode=0
Sep 15 11:44:28.219610 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:28.219382    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm" event={"ID":"ff8be1d6-0235-434a-8c16-a35a3769423a","Type":"ContainerDied","Data":"30f0d05f5f6c9fa81c388fdc466011a0093a9e9e4bde15510b95dd578fff468a"}
Sep 15 11:44:29.230014 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:29.229981    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff8be1d6-0235-434a-8c16-a35a3769423a" containerID="3c9f664fc9c1a66eb21552f454fad928f79065ed88af020e7fe81f68111ab75a" exitCode=0
Sep 15 11:44:29.230369 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:29.230071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm" event={"ID":"ff8be1d6-0235-434a-8c16-a35a3769423a","Type":"ContainerDied","Data":"3c9f664fc9c1a66eb21552f454fad928f79065ed88af020e7fe81f68111ab75a"}
Sep 15 11:44:30.355455 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:30.355432    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"
Sep 15 11:44:30.444437 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:30.444407    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr44w\" (UniqueName: \"kubernetes.io/projected/ff8be1d6-0235-434a-8c16-a35a3769423a-kube-api-access-qr44w\") pod \"ff8be1d6-0235-434a-8c16-a35a3769423a\" (UID: \"ff8be1d6-0235-434a-8c16-a35a3769423a\") "
Sep 15 11:44:30.444598 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:30.444482    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff8be1d6-0235-434a-8c16-a35a3769423a-util\") pod \"ff8be1d6-0235-434a-8c16-a35a3769423a\" (UID: \"ff8be1d6-0235-434a-8c16-a35a3769423a\") "
Sep 15 11:44:30.444598 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:30.444509    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff8be1d6-0235-434a-8c16-a35a3769423a-bundle\") pod \"ff8be1d6-0235-434a-8c16-a35a3769423a\" (UID: \"ff8be1d6-0235-434a-8c16-a35a3769423a\") "
Sep 15 11:44:30.444922 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:30.444899    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8be1d6-0235-434a-8c16-a35a3769423a-bundle" (OuterVolumeSpecName: "bundle") pod "ff8be1d6-0235-434a-8c16-a35a3769423a" (UID: "ff8be1d6-0235-434a-8c16-a35a3769423a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:44:30.446597 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:30.446580    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8be1d6-0235-434a-8c16-a35a3769423a-kube-api-access-qr44w" (OuterVolumeSpecName: "kube-api-access-qr44w") pod "ff8be1d6-0235-434a-8c16-a35a3769423a" (UID: "ff8be1d6-0235-434a-8c16-a35a3769423a"). InnerVolumeSpecName "kube-api-access-qr44w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:44:30.449865 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:30.449846    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8be1d6-0235-434a-8c16-a35a3769423a-util" (OuterVolumeSpecName: "util") pod "ff8be1d6-0235-434a-8c16-a35a3769423a" (UID: "ff8be1d6-0235-434a-8c16-a35a3769423a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:44:30.545753 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:30.545696    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff8be1d6-0235-434a-8c16-a35a3769423a-bundle\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:44:30.545753 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:30.545718    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr44w\" (UniqueName: \"kubernetes.io/projected/ff8be1d6-0235-434a-8c16-a35a3769423a-kube-api-access-qr44w\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:44:30.545753 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:30.545729    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff8be1d6-0235-434a-8c16-a35a3769423a-util\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:44:31.239884 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.239841    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm"
Sep 15 11:44:31.239884 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.239849    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qgwm" event={"ID":"ff8be1d6-0235-434a-8c16-a35a3769423a","Type":"ContainerDied","Data":"6a0732541c1114a1333c344e31e3ebf19fb91659bef6292cf323f437957b9a4f"}
Sep 15 11:44:31.240072 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.239904    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a0732541c1114a1333c344e31e3ebf19fb91659bef6292cf323f437957b9a4f"
Sep 15 11:44:31.425159 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.425127    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-td87g"]
Sep 15 11:44:31.425492 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.425419    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff8be1d6-0235-434a-8c16-a35a3769423a" containerName="pull"
Sep 15 11:44:31.425492 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.425431    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff8be1d6-0235-434a-8c16-a35a3769423a" containerName="pull"
Sep 15 11:44:31.425492 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.425441    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff8be1d6-0235-434a-8c16-a35a3769423a" containerName="util"
Sep 15 11:44:31.425492 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.425446    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff8be1d6-0235-434a-8c16-a35a3769423a" containerName="util"
Sep 15 11:44:31.425492 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.425452    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff8be1d6-0235-434a-8c16-a35a3769423a" containerName="extract"
Sep 15 11:44:31.425492 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.425461    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff8be1d6-0235-434a-8c16-a35a3769423a" containerName="extract"
Sep 15 11:44:31.425711 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.425519    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff8be1d6-0235-434a-8c16-a35a3769423a" containerName="extract"
Sep 15 11:44:31.428368 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.428352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-td87g"
Sep 15 11:44:31.430542 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.430519    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5zcrl\""
Sep 15 11:44:31.430644 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.430563    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 11:44:31.430644 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.430563    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 11:44:31.438395 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.438374    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-td87g"]
Sep 15 11:44:31.552319 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.552236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd6bd0c6-3111-4606-b2e9-d88cb94c5012-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-td87g\" (UID: \"fd6bd0c6-3111-4606-b2e9-d88cb94c5012\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-td87g"
Sep 15 11:44:31.552319 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.552272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g7c7\" (UniqueName: \"kubernetes.io/projected/fd6bd0c6-3111-4606-b2e9-d88cb94c5012-kube-api-access-5g7c7\") pod \"cert-manager-cainjector-559cdd8c87-td87g\" (UID: \"fd6bd0c6-3111-4606-b2e9-d88cb94c5012\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-td87g"
Sep 15 11:44:31.653142 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.653116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd6bd0c6-3111-4606-b2e9-d88cb94c5012-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-td87g\" (UID: \"fd6bd0c6-3111-4606-b2e9-d88cb94c5012\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-td87g"
Sep 15 11:44:31.653242 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.653146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g7c7\" (UniqueName: \"kubernetes.io/projected/fd6bd0c6-3111-4606-b2e9-d88cb94c5012-kube-api-access-5g7c7\") pod \"cert-manager-cainjector-559cdd8c87-td87g\" (UID: \"fd6bd0c6-3111-4606-b2e9-d88cb94c5012\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-td87g"
Sep 15 11:44:31.661839 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.661813    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 11:44:31.662148 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.662130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd6bd0c6-3111-4606-b2e9-d88cb94c5012-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-td87g\" (UID: \"fd6bd0c6-3111-4606-b2e9-d88cb94c5012\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-td87g"
Sep 15 11:44:31.673096 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.673075    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 11:44:31.683651 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.683629    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g7c7\" (UniqueName: \"kubernetes.io/projected/fd6bd0c6-3111-4606-b2e9-d88cb94c5012-kube-api-access-5g7c7\") pod \"cert-manager-cainjector-559cdd8c87-td87g\" (UID: \"fd6bd0c6-3111-4606-b2e9-d88cb94c5012\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-td87g"
Sep 15 11:44:31.739549 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.739524    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5zcrl\""
Sep 15 11:44:31.748270 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:31.748251    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-td87g"
Sep 15 11:44:32.020607 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:32.020469    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-td87g"]
Sep 15 11:44:32.023016 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:44:32.022990    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd6bd0c6_3111_4606_b2e9_d88cb94c5012.slice/crio-3b75b61401730fe6b004c8a8dc0df2af5309f59f9da31256feeb71f42e1b43be WatchSource:0}: Error finding container 3b75b61401730fe6b004c8a8dc0df2af5309f59f9da31256feeb71f42e1b43be: Status 404 returned error can't find the container with id 3b75b61401730fe6b004c8a8dc0df2af5309f59f9da31256feeb71f42e1b43be
Sep 15 11:44:32.245056 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:32.245019    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-td87g" event={"ID":"fd6bd0c6-3111-4606-b2e9-d88cb94c5012","Type":"ContainerStarted","Data":"3b75b61401730fe6b004c8a8dc0df2af5309f59f9da31256feeb71f42e1b43be"}
Sep 15 11:44:34.737760 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:34.737723    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k5pms"]
Sep 15 11:44:34.741135 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:34.741112    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-k5pms"
Sep 15 11:44:34.743108 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:34.743082    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ntppv\""
Sep 15 11:44:34.751409 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:34.751385    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k5pms"]
Sep 15 11:44:34.775625 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:34.775593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9kc\" (UniqueName: \"kubernetes.io/projected/335e0ece-5bc8-4bf2-af4d-c1333353cc84-kube-api-access-9b9kc\") pod \"cert-manager-55b4d954d5-k5pms\" (UID: \"335e0ece-5bc8-4bf2-af4d-c1333353cc84\") " pod="cert-manager/cert-manager-55b4d954d5-k5pms"
Sep 15 11:44:34.775766 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:34.775737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/335e0ece-5bc8-4bf2-af4d-c1333353cc84-bound-sa-token\") pod \"cert-manager-55b4d954d5-k5pms\" (UID: \"335e0ece-5bc8-4bf2-af4d-c1333353cc84\") " pod="cert-manager/cert-manager-55b4d954d5-k5pms"
Sep 15 11:44:34.877160 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:34.877122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/335e0ece-5bc8-4bf2-af4d-c1333353cc84-bound-sa-token\") pod \"cert-manager-55b4d954d5-k5pms\" (UID: \"335e0ece-5bc8-4bf2-af4d-c1333353cc84\") " pod="cert-manager/cert-manager-55b4d954d5-k5pms"
Sep 15 11:44:34.877313 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:34.877181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9kc\" (UniqueName: \"kubernetes.io/projected/335e0ece-5bc8-4bf2-af4d-c1333353cc84-kube-api-access-9b9kc\") pod \"cert-manager-55b4d954d5-k5pms\" (UID: \"335e0ece-5bc8-4bf2-af4d-c1333353cc84\") " pod="cert-manager/cert-manager-55b4d954d5-k5pms"
Sep 15 11:44:34.895347 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:34.895318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/335e0ece-5bc8-4bf2-af4d-c1333353cc84-bound-sa-token\") pod \"cert-manager-55b4d954d5-k5pms\" (UID: \"335e0ece-5bc8-4bf2-af4d-c1333353cc84\") " pod="cert-manager/cert-manager-55b4d954d5-k5pms"
Sep 15 11:44:34.895456 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:34.895440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9kc\" (UniqueName: \"kubernetes.io/projected/335e0ece-5bc8-4bf2-af4d-c1333353cc84-kube-api-access-9b9kc\") pod \"cert-manager-55b4d954d5-k5pms\" (UID: \"335e0ece-5bc8-4bf2-af4d-c1333353cc84\") " pod="cert-manager/cert-manager-55b4d954d5-k5pms"
Sep 15 11:44:35.053449 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:35.053365    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-k5pms"
Sep 15 11:44:35.264581 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:35.264543    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-td87g" event={"ID":"fd6bd0c6-3111-4606-b2e9-d88cb94c5012","Type":"ContainerStarted","Data":"6043fdfd46fa474fc5a76d9cdb3b8dd9e173e5f31a1b5fe45467919d862413bc"}
Sep 15 11:44:35.283416 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:35.283359    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-td87g" podStartSLOduration=1.414963954 podStartE2EDuration="4.283339757s" totalImagesPullingTime="2.868375803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:44:35.280720966 +0000 UTC m=+364.061292355" watchObservedRunningTime="2026-09-15 11:44:35.283339757 +0000 UTC m=+364.063911146"
Sep 15 11:44:35.325334 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:35.325307    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k5pms"]
Sep 15 11:44:35.327934 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:44:35.327903    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod335e0ece_5bc8_4bf2_af4d_c1333353cc84.slice/crio-1e9ed23d19b77d98301f792fae8c542e8cd65f6ee97fa2d2cca136d6f4603c42 WatchSource:0}: Error finding container 1e9ed23d19b77d98301f792fae8c542e8cd65f6ee97fa2d2cca136d6f4603c42: Status 404 returned error can't find the container with id 1e9ed23d19b77d98301f792fae8c542e8cd65f6ee97fa2d2cca136d6f4603c42
Sep 15 11:44:36.269883 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:36.269842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-k5pms" event={"ID":"335e0ece-5bc8-4bf2-af4d-c1333353cc84","Type":"ContainerStarted","Data":"06065e274dda751e9a6313935fa27bdee3ff88233305a0905daba3f0539d437d"}
Sep 15 11:44:36.270247 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:36.269908    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-k5pms" event={"ID":"335e0ece-5bc8-4bf2-af4d-c1333353cc84","Type":"ContainerStarted","Data":"1e9ed23d19b77d98301f792fae8c542e8cd65f6ee97fa2d2cca136d6f4603c42"}
Sep 15 11:44:36.286931 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:36.286862    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-k5pms" podStartSLOduration=2.286846214 podStartE2EDuration="2.286846214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:44:36.286389909 +0000 UTC m=+365.066961301" watchObservedRunningTime="2026-09-15 11:44:36.286846214 +0000 UTC m=+365.067417604"
Sep 15 11:44:45.705139 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.705104    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"]
Sep 15 11:44:45.726747 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.726719    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"
Sep 15 11:44:45.729504 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.729483    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 11:44:45.730293 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.730273    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bzsrv\""
Sep 15 11:44:45.730404 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.730300    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 11:44:45.732610 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.732587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"]
Sep 15 11:44:45.858474 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.858437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb25d298-e774-417b-bcf1-afa5d5e1edf1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl\" (UID: \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"
Sep 15 11:44:45.858616 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.858506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb25d298-e774-417b-bcf1-afa5d5e1edf1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl\" (UID: \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"
Sep 15 11:44:45.858616 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.858552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-459dn\" (UniqueName: \"kubernetes.io/projected/fb25d298-e774-417b-bcf1-afa5d5e1edf1-kube-api-access-459dn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl\" (UID: \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"
Sep 15 11:44:45.959510 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.959425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb25d298-e774-417b-bcf1-afa5d5e1edf1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl\" (UID: \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"
Sep 15 11:44:45.959510 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.959464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-459dn\" (UniqueName: \"kubernetes.io/projected/fb25d298-e774-417b-bcf1-afa5d5e1edf1-kube-api-access-459dn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl\" (UID: \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"
Sep 15 11:44:45.959510 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.959498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb25d298-e774-417b-bcf1-afa5d5e1edf1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl\" (UID: \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"
Sep 15 11:44:45.959966 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.959945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb25d298-e774-417b-bcf1-afa5d5e1edf1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl\" (UID: \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"
Sep 15 11:44:45.959966 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.959953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb25d298-e774-417b-bcf1-afa5d5e1edf1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl\" (UID: \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"
Sep 15 11:44:45.968451 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:45.968431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-459dn\" (UniqueName: \"kubernetes.io/projected/fb25d298-e774-417b-bcf1-afa5d5e1edf1-kube-api-access-459dn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl\" (UID: \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"
Sep 15 11:44:46.037492 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:46.037457    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"
Sep 15 11:44:46.324681 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:46.324653    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"]
Sep 15 11:44:46.327128 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:44:46.327103    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb25d298_e774_417b_bcf1_afa5d5e1edf1.slice/crio-3147b660efd5a3703e631542ba25b84a143d420156f9be2c6688ecb5247720df WatchSource:0}: Error finding container 3147b660efd5a3703e631542ba25b84a143d420156f9be2c6688ecb5247720df: Status 404 returned error can't find the container with id 3147b660efd5a3703e631542ba25b84a143d420156f9be2c6688ecb5247720df
Sep 15 11:44:47.313675 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:47.313582    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb25d298-e774-417b-bcf1-afa5d5e1edf1" containerID="80ae57988e9b7ee5cf53d750fff40accf48a1ce8202ca82a264221a1439ab68e" exitCode=0
Sep 15 11:44:47.313675 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:47.313637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl" event={"ID":"fb25d298-e774-417b-bcf1-afa5d5e1edf1","Type":"ContainerDied","Data":"80ae57988e9b7ee5cf53d750fff40accf48a1ce8202ca82a264221a1439ab68e"}
Sep 15 11:44:47.314229 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:47.313678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl" event={"ID":"fb25d298-e774-417b-bcf1-afa5d5e1edf1","Type":"ContainerStarted","Data":"3147b660efd5a3703e631542ba25b84a143d420156f9be2c6688ecb5247720df"}
Sep 15 11:44:48.319898 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:48.319840    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl" event={"ID":"fb25d298-e774-417b-bcf1-afa5d5e1edf1","Type":"ContainerStarted","Data":"a9dd4c1d3c96f2325be4ee053fa7522ebe04d44c0837881e37e41cb7f5a0391a"}
Sep 15 11:44:49.325792 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:49.325759    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb25d298-e774-417b-bcf1-afa5d5e1edf1" containerID="a9dd4c1d3c96f2325be4ee053fa7522ebe04d44c0837881e37e41cb7f5a0391a" exitCode=0
Sep 15 11:44:49.326193 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:49.325851    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl" event={"ID":"fb25d298-e774-417b-bcf1-afa5d5e1edf1","Type":"ContainerDied","Data":"a9dd4c1d3c96f2325be4ee053fa7522ebe04d44c0837881e37e41cb7f5a0391a"}
Sep 15 11:44:50.332464 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:50.332425    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb25d298-e774-417b-bcf1-afa5d5e1edf1" containerID="fd083e48fb92d3519f86dd9e7104f9ba72d84e513399e329e6bc191765cf04b2" exitCode=0
Sep 15 11:44:50.332827 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:50.332491    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl" event={"ID":"fb25d298-e774-417b-bcf1-afa5d5e1edf1","Type":"ContainerDied","Data":"fd083e48fb92d3519f86dd9e7104f9ba72d84e513399e329e6bc191765cf04b2"}
Sep 15 11:44:51.452230 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:51.452206    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"
Sep 15 11:44:51.603193 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:51.603114    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb25d298-e774-417b-bcf1-afa5d5e1edf1-util\") pod \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\" (UID: \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\") "
Sep 15 11:44:51.603193 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:51.603176    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb25d298-e774-417b-bcf1-afa5d5e1edf1-bundle\") pod \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\" (UID: \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\") "
Sep 15 11:44:51.603360 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:51.603206    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-459dn\" (UniqueName: \"kubernetes.io/projected/fb25d298-e774-417b-bcf1-afa5d5e1edf1-kube-api-access-459dn\") pod \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\" (UID: \"fb25d298-e774-417b-bcf1-afa5d5e1edf1\") "
Sep 15 11:44:51.603914 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:51.603860    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb25d298-e774-417b-bcf1-afa5d5e1edf1-bundle" (OuterVolumeSpecName: "bundle") pod "fb25d298-e774-417b-bcf1-afa5d5e1edf1" (UID: "fb25d298-e774-417b-bcf1-afa5d5e1edf1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:44:51.605335 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:51.605312    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb25d298-e774-417b-bcf1-afa5d5e1edf1-kube-api-access-459dn" (OuterVolumeSpecName: "kube-api-access-459dn") pod "fb25d298-e774-417b-bcf1-afa5d5e1edf1" (UID: "fb25d298-e774-417b-bcf1-afa5d5e1edf1"). InnerVolumeSpecName "kube-api-access-459dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:44:51.608768 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:51.608748    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb25d298-e774-417b-bcf1-afa5d5e1edf1-util" (OuterVolumeSpecName: "util") pod "fb25d298-e774-417b-bcf1-afa5d5e1edf1" (UID: "fb25d298-e774-417b-bcf1-afa5d5e1edf1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:44:51.704260 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:51.704230    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb25d298-e774-417b-bcf1-afa5d5e1edf1-bundle\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:44:51.704260 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:51.704256    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-459dn\" (UniqueName: \"kubernetes.io/projected/fb25d298-e774-417b-bcf1-afa5d5e1edf1-kube-api-access-459dn\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:44:51.704260 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:51.704266    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb25d298-e774-417b-bcf1-afa5d5e1edf1-util\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:44:52.342404 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:52.342372    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl"
Sep 15 11:44:52.342404 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:52.342386    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgs2vl" event={"ID":"fb25d298-e774-417b-bcf1-afa5d5e1edf1","Type":"ContainerDied","Data":"3147b660efd5a3703e631542ba25b84a143d420156f9be2c6688ecb5247720df"}
Sep 15 11:44:52.342646 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:44:52.342427    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3147b660efd5a3703e631542ba25b84a143d420156f9be2c6688ecb5247720df"
Sep 15 11:45:00.940047 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:00.940013    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"]
Sep 15 11:45:00.940515 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:00.940287    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb25d298-e774-417b-bcf1-afa5d5e1edf1" containerName="util"
Sep 15 11:45:00.940515 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:00.940299    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb25d298-e774-417b-bcf1-afa5d5e1edf1" containerName="util"
Sep 15 11:45:00.940515 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:00.940309    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb25d298-e774-417b-bcf1-afa5d5e1edf1" containerName="pull"
Sep 15 11:45:00.940515 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:00.940314    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb25d298-e774-417b-bcf1-afa5d5e1edf1" containerName="pull"
Sep 15 11:45:00.940515 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:00.940330    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb25d298-e774-417b-bcf1-afa5d5e1edf1" containerName="extract"
Sep 15 11:45:00.940515 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:00.940339    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb25d298-e774-417b-bcf1-afa5d5e1edf1" containerName="extract"
Sep 15 11:45:00.940515 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:00.940396    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb25d298-e774-417b-bcf1-afa5d5e1edf1" containerName="extract"
Sep 15 11:45:00.947797 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:00.947775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"
Sep 15 11:45:00.950519 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:00.950491    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bzsrv\""
Sep 15 11:45:00.951573 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:00.951547    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 11:45:00.951694 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:00.951667    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 11:45:00.953555 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:00.953496    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"]
Sep 15 11:45:01.078112 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:01.078058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b001ebe1-4a30-4646-9ea5-baa01a15a65f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8\" (UID: \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"
Sep 15 11:45:01.078316 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:01.078125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b001ebe1-4a30-4646-9ea5-baa01a15a65f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8\" (UID: \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"
Sep 15 11:45:01.078316 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:01.078164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wv55\" (UniqueName: \"kubernetes.io/projected/b001ebe1-4a30-4646-9ea5-baa01a15a65f-kube-api-access-2wv55\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8\" (UID: \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"
Sep 15 11:45:01.179045 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:01.179007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b001ebe1-4a30-4646-9ea5-baa01a15a65f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8\" (UID: \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"
Sep 15 11:45:01.179045 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:01.179050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b001ebe1-4a30-4646-9ea5-baa01a15a65f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8\" (UID: \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"
Sep 15 11:45:01.179298 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:01.179074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wv55\" (UniqueName: \"kubernetes.io/projected/b001ebe1-4a30-4646-9ea5-baa01a15a65f-kube-api-access-2wv55\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8\" (UID: \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"
Sep 15 11:45:01.179413 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:01.179392    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b001ebe1-4a30-4646-9ea5-baa01a15a65f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8\" (UID: \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"
Sep 15 11:45:01.179484 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:01.179461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b001ebe1-4a30-4646-9ea5-baa01a15a65f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8\" (UID: \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"
Sep 15 11:45:01.187965 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:01.187936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wv55\" (UniqueName: \"kubernetes.io/projected/b001ebe1-4a30-4646-9ea5-baa01a15a65f-kube-api-access-2wv55\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8\" (UID: \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"
Sep 15 11:45:01.262984 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:01.262862    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"
Sep 15 11:45:01.548994 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:01.548968    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"]
Sep 15 11:45:01.551339 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:45:01.551305    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb001ebe1_4a30_4646_9ea5_baa01a15a65f.slice/crio-c1939792b4462593042d1d16e51be6ec1958e7901a4c90a9624af644378ac8dc WatchSource:0}: Error finding container c1939792b4462593042d1d16e51be6ec1958e7901a4c90a9624af644378ac8dc: Status 404 returned error can't find the container with id c1939792b4462593042d1d16e51be6ec1958e7901a4c90a9624af644378ac8dc
Sep 15 11:45:02.383727 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:02.383683    2580 generic.go:356] "Generic (PLEG): container finished" podID="b001ebe1-4a30-4646-9ea5-baa01a15a65f" containerID="6805d2fd329a7136139b5a4d0887b8e3c7fbaa129bc8e0f246c14e97ca89bd0d" exitCode=0
Sep 15 11:45:02.384216 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:02.383771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8" event={"ID":"b001ebe1-4a30-4646-9ea5-baa01a15a65f","Type":"ContainerDied","Data":"6805d2fd329a7136139b5a4d0887b8e3c7fbaa129bc8e0f246c14e97ca89bd0d"}
Sep 15 11:45:02.384216 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:02.383823    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8" event={"ID":"b001ebe1-4a30-4646-9ea5-baa01a15a65f","Type":"ContainerStarted","Data":"c1939792b4462593042d1d16e51be6ec1958e7901a4c90a9624af644378ac8dc"}
Sep 15 11:45:03.057984 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.057952    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"]
Sep 15 11:45:03.061243 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.061225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.064736 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.064715    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 11:45:03.065110 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.065088    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bx677\""
Sep 15 11:45:03.065110 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.065101    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 11:45:03.065255 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.065101    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 11:45:03.065255 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.065101    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 11:45:03.074378 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.074355    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"]
Sep 15 11:45:03.192655 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.192622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de62f8dd-efac-420e-ae3b-e67de86faad7-tmp\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.192655 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.192657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de62f8dd-efac-420e-ae3b-e67de86faad7-manifests\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.192930 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.192677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de62f8dd-efac-420e-ae3b-e67de86faad7-webhook-cert\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.192930 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.192753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg9kx\" (UniqueName: \"kubernetes.io/projected/de62f8dd-efac-420e-ae3b-e67de86faad7-kube-api-access-qg9kx\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.192930 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.192812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de62f8dd-efac-420e-ae3b-e67de86faad7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.293658 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.293619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de62f8dd-efac-420e-ae3b-e67de86faad7-webhook-cert\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.293849 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.293677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg9kx\" (UniqueName: \"kubernetes.io/projected/de62f8dd-efac-420e-ae3b-e67de86faad7-kube-api-access-qg9kx\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.293849 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.293788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de62f8dd-efac-420e-ae3b-e67de86faad7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.294017 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.293904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de62f8dd-efac-420e-ae3b-e67de86faad7-tmp\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.294017 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.293937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de62f8dd-efac-420e-ae3b-e67de86faad7-manifests\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.294337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.294312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de62f8dd-efac-420e-ae3b-e67de86faad7-manifests\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.296086 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.296055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de62f8dd-efac-420e-ae3b-e67de86faad7-tmp\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.296271 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.296251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de62f8dd-efac-420e-ae3b-e67de86faad7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.296380 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.296357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de62f8dd-efac-420e-ae3b-e67de86faad7-webhook-cert\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.304627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.304599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg9kx\" (UniqueName: \"kubernetes.io/projected/de62f8dd-efac-420e-ae3b-e67de86faad7-kube-api-access-qg9kx\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-2f7rq\" (UID: \"de62f8dd-efac-420e-ae3b-e67de86faad7\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.370565 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.370533    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:03.391529 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.391496    2580 generic.go:356] "Generic (PLEG): container finished" podID="b001ebe1-4a30-4646-9ea5-baa01a15a65f" containerID="383d1f771dcad30e7b4a22c7da84daa12c4ad11fb73420cfaeb338be68231292" exitCode=0
Sep 15 11:45:03.391937 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.391593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8" event={"ID":"b001ebe1-4a30-4646-9ea5-baa01a15a65f","Type":"ContainerDied","Data":"383d1f771dcad30e7b4a22c7da84daa12c4ad11fb73420cfaeb338be68231292"}
Sep 15 11:45:03.651388 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:03.651365    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"]
Sep 15 11:45:03.653936 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:45:03.653909    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde62f8dd_efac_420e_ae3b_e67de86faad7.slice/crio-ec515d9f8439e5672f746e8855d0a92a539e3d2e6f35b2c59b164bb9d7f10b42 WatchSource:0}: Error finding container ec515d9f8439e5672f746e8855d0a92a539e3d2e6f35b2c59b164bb9d7f10b42: Status 404 returned error can't find the container with id ec515d9f8439e5672f746e8855d0a92a539e3d2e6f35b2c59b164bb9d7f10b42
Sep 15 11:45:04.400064 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:04.400022    2580 generic.go:356] "Generic (PLEG): container finished" podID="b001ebe1-4a30-4646-9ea5-baa01a15a65f" containerID="be5f235e2d663c416f46322bb4f84c47db299d88488df934a5fb432428d6af99" exitCode=0
Sep 15 11:45:04.400524 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:04.400114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8" event={"ID":"b001ebe1-4a30-4646-9ea5-baa01a15a65f","Type":"ContainerDied","Data":"be5f235e2d663c416f46322bb4f84c47db299d88488df934a5fb432428d6af99"}
Sep 15 11:45:04.401853 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:04.401795    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq" event={"ID":"de62f8dd-efac-420e-ae3b-e67de86faad7","Type":"ContainerStarted","Data":"ec515d9f8439e5672f746e8855d0a92a539e3d2e6f35b2c59b164bb9d7f10b42"}
Sep 15 11:45:06.617891 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:06.617853    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"
Sep 15 11:45:06.724003 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:06.723977    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b001ebe1-4a30-4646-9ea5-baa01a15a65f-bundle\") pod \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\" (UID: \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\") "
Sep 15 11:45:06.724142 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:06.724018    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b001ebe1-4a30-4646-9ea5-baa01a15a65f-util\") pod \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\" (UID: \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\") "
Sep 15 11:45:06.724142 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:06.724086    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wv55\" (UniqueName: \"kubernetes.io/projected/b001ebe1-4a30-4646-9ea5-baa01a15a65f-kube-api-access-2wv55\") pod \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\" (UID: \"b001ebe1-4a30-4646-9ea5-baa01a15a65f\") "
Sep 15 11:45:06.724859 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:06.724835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b001ebe1-4a30-4646-9ea5-baa01a15a65f-bundle" (OuterVolumeSpecName: "bundle") pod "b001ebe1-4a30-4646-9ea5-baa01a15a65f" (UID: "b001ebe1-4a30-4646-9ea5-baa01a15a65f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:45:06.727400 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:06.727375    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b001ebe1-4a30-4646-9ea5-baa01a15a65f-kube-api-access-2wv55" (OuterVolumeSpecName: "kube-api-access-2wv55") pod "b001ebe1-4a30-4646-9ea5-baa01a15a65f" (UID: "b001ebe1-4a30-4646-9ea5-baa01a15a65f"). InnerVolumeSpecName "kube-api-access-2wv55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:45:06.791160 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:06.791125    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b001ebe1-4a30-4646-9ea5-baa01a15a65f-util" (OuterVolumeSpecName: "util") pod "b001ebe1-4a30-4646-9ea5-baa01a15a65f" (UID: "b001ebe1-4a30-4646-9ea5-baa01a15a65f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:45:06.824991 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:06.824962    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wv55\" (UniqueName: \"kubernetes.io/projected/b001ebe1-4a30-4646-9ea5-baa01a15a65f-kube-api-access-2wv55\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:45:06.824991 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:06.824990    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b001ebe1-4a30-4646-9ea5-baa01a15a65f-bundle\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:45:06.825134 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:06.824999    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b001ebe1-4a30-4646-9ea5-baa01a15a65f-util\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:45:07.416025 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:07.415984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq" event={"ID":"de62f8dd-efac-420e-ae3b-e67de86faad7","Type":"ContainerStarted","Data":"f0e9b462f23748c30363332db0b4640fcf8cbbda50663bccf64cd835ac72ef4e"}
Sep 15 11:45:07.417922 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:07.417901    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8"
Sep 15 11:45:07.418075 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:07.417920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kl6m8" event={"ID":"b001ebe1-4a30-4646-9ea5-baa01a15a65f","Type":"ContainerDied","Data":"c1939792b4462593042d1d16e51be6ec1958e7901a4c90a9624af644378ac8dc"}
Sep 15 11:45:07.418075 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:07.417960    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1939792b4462593042d1d16e51be6ec1958e7901a4c90a9624af644378ac8dc"
Sep 15 11:45:08.424176 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:08.424139    2580 generic.go:356] "Generic (PLEG): container finished" podID="de62f8dd-efac-420e-ae3b-e67de86faad7" containerID="f0e9b462f23748c30363332db0b4640fcf8cbbda50663bccf64cd835ac72ef4e" exitCode=0
Sep 15 11:45:08.424642 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:08.424221    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq" event={"ID":"de62f8dd-efac-420e-ae3b-e67de86faad7","Type":"ContainerDied","Data":"f0e9b462f23748c30363332db0b4640fcf8cbbda50663bccf64cd835ac72ef4e"}
Sep 15 11:45:09.431771 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:09.431740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq" event={"ID":"de62f8dd-efac-420e-ae3b-e67de86faad7","Type":"ContainerStarted","Data":"fe10c8fc6963e1c7f63803a0594025f3356b22895f3d2fc682ae6721bec7e979"}
Sep 15 11:45:09.432230 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:09.431858    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:09.458957 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:09.458902    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq" podStartSLOduration=1.9017150900000002 podStartE2EDuration="6.458867355s" totalImagesPullingTime="3.515206125s" totalInitContainerRuntime="1.04194614s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:09.458691894 +0000 UTC m=+398.239263286" watchObservedRunningTime="2026-09-15 11:45:09.458867355 +0000 UTC m=+398.239438743"
Sep 15 11:45:20.438821 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:20.438784    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-2f7rq"
Sep 15 11:45:22.722699 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.722659    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp"]
Sep 15 11:45:22.723184 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.723090    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b001ebe1-4a30-4646-9ea5-baa01a15a65f" containerName="pull"
Sep 15 11:45:22.723184 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.723109    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b001ebe1-4a30-4646-9ea5-baa01a15a65f" containerName="pull"
Sep 15 11:45:22.723184 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.723123    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b001ebe1-4a30-4646-9ea5-baa01a15a65f" containerName="extract"
Sep 15 11:45:22.723184 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.723131    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b001ebe1-4a30-4646-9ea5-baa01a15a65f" containerName="extract"
Sep 15 11:45:22.723184 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.723150    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b001ebe1-4a30-4646-9ea5-baa01a15a65f" containerName="util"
Sep 15 11:45:22.723184 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.723158    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b001ebe1-4a30-4646-9ea5-baa01a15a65f" containerName="util"
Sep 15 11:45:22.723499 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.723226    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b001ebe1-4a30-4646-9ea5-baa01a15a65f" containerName="extract"
Sep 15 11:45:22.731747 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.731718    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp"
Sep 15 11:45:22.734467 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.734437    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 11:45:22.735857 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.735829    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 11:45:22.736168 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.736114    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hm6dh\""
Sep 15 11:45:22.739754 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.736827    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 11:45:22.739754 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.739084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp"]
Sep 15 11:45:22.741842 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.741820    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 11:45:22.748769 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.748750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dba8ee1-dd2e-429c-9585-3411f89d407b-tls-certs\") pod \"kube-auth-proxy-df8dfc748-6hsvp\" (UID: \"6dba8ee1-dd2e-429c-9585-3411f89d407b\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp"
Sep 15 11:45:22.748851 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.748805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dba8ee1-dd2e-429c-9585-3411f89d407b-tmp\") pod \"kube-auth-proxy-df8dfc748-6hsvp\" (UID: \"6dba8ee1-dd2e-429c-9585-3411f89d407b\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp"
Sep 15 11:45:22.748948 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.748912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjxp\" (UniqueName: \"kubernetes.io/projected/6dba8ee1-dd2e-429c-9585-3411f89d407b-kube-api-access-5wjxp\") pod \"kube-auth-proxy-df8dfc748-6hsvp\" (UID: \"6dba8ee1-dd2e-429c-9585-3411f89d407b\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp"
Sep 15 11:45:22.849613 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.849581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dba8ee1-dd2e-429c-9585-3411f89d407b-tmp\") pod \"kube-auth-proxy-df8dfc748-6hsvp\" (UID: \"6dba8ee1-dd2e-429c-9585-3411f89d407b\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp"
Sep 15 11:45:22.849760 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.849642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjxp\" (UniqueName: \"kubernetes.io/projected/6dba8ee1-dd2e-429c-9585-3411f89d407b-kube-api-access-5wjxp\") pod \"kube-auth-proxy-df8dfc748-6hsvp\" (UID: \"6dba8ee1-dd2e-429c-9585-3411f89d407b\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp"
Sep 15 11:45:22.849760 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.849690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dba8ee1-dd2e-429c-9585-3411f89d407b-tls-certs\") pod \"kube-auth-proxy-df8dfc748-6hsvp\" (UID: \"6dba8ee1-dd2e-429c-9585-3411f89d407b\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp"
Sep 15 11:45:22.851939 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.851911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dba8ee1-dd2e-429c-9585-3411f89d407b-tmp\") pod \"kube-auth-proxy-df8dfc748-6hsvp\" (UID: \"6dba8ee1-dd2e-429c-9585-3411f89d407b\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp"
Sep 15 11:45:22.852047 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.852029    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dba8ee1-dd2e-429c-9585-3411f89d407b-tls-certs\") pod \"kube-auth-proxy-df8dfc748-6hsvp\" (UID: \"6dba8ee1-dd2e-429c-9585-3411f89d407b\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp"
Sep 15 11:45:22.858303 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:22.858269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjxp\" (UniqueName: \"kubernetes.io/projected/6dba8ee1-dd2e-429c-9585-3411f89d407b-kube-api-access-5wjxp\") pod \"kube-auth-proxy-df8dfc748-6hsvp\" (UID: \"6dba8ee1-dd2e-429c-9585-3411f89d407b\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp"
Sep 15 11:45:23.047737 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:23.047663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp"
Sep 15 11:45:23.333789 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:23.333763    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp"]
Sep 15 11:45:23.336080 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:45:23.336048    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dba8ee1_dd2e_429c_9585_3411f89d407b.slice/crio-580459794a62b7b50f3cb5017bba7f33dc7f03f99f4f40bbd5ee9b8aa5951f92 WatchSource:0}: Error finding container 580459794a62b7b50f3cb5017bba7f33dc7f03f99f4f40bbd5ee9b8aa5951f92: Status 404 returned error can't find the container with id 580459794a62b7b50f3cb5017bba7f33dc7f03f99f4f40bbd5ee9b8aa5951f92
Sep 15 11:45:23.495005 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:23.494971    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp" event={"ID":"6dba8ee1-dd2e-429c-9585-3411f89d407b","Type":"ContainerStarted","Data":"580459794a62b7b50f3cb5017bba7f33dc7f03f99f4f40bbd5ee9b8aa5951f92"}
Sep 15 11:45:26.513101 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:26.513073    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp" event={"ID":"6dba8ee1-dd2e-429c-9585-3411f89d407b","Type":"ContainerStarted","Data":"ff22d28db954916c198f8c0eb06b71adb8c40f8d13343589942a66b8ff93079a"}
Sep 15 11:45:26.531656 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:26.531608    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-df8dfc748-6hsvp" podStartSLOduration=1.427672413 podStartE2EDuration="4.531591774s" totalImagesPullingTime="3.103919361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:26.529599402 +0000 UTC m=+415.310170791" watchObservedRunningTime="2026-09-15 11:45:26.531591774 +0000 UTC m=+415.312163162"
Sep 15 11:45:28.727148 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.727107    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"]
Sep 15 11:45:28.731494 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.731466    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.733913 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.733892    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 11:45:28.734034 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.733925    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-szvjj\""
Sep 15 11:45:28.739516 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.739493    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"]
Sep 15 11:45:28.791650 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.791625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6930cf41-976c-4b06-a522-9fc7f633e8e0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.791782 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.791656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.791782 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.791676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.791782 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.791731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6930cf41-976c-4b06-a522-9fc7f633e8e0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.791782 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.791773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6930cf41-976c-4b06-a522-9fc7f633e8e0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.791944 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.791819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxn8\" (UniqueName: \"kubernetes.io/projected/6930cf41-976c-4b06-a522-9fc7f633e8e0-kube-api-access-6zxn8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.791944 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.791839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.791944 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.791900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.791944 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.791934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.892280 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.892253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.892280 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.892283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.892497 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.892309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6930cf41-976c-4b06-a522-9fc7f633e8e0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.892497 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.892325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.892497 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.892346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.892497 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.892473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6930cf41-976c-4b06-a522-9fc7f633e8e0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.892667 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.892515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6930cf41-976c-4b06-a522-9fc7f633e8e0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.892723 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.892691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxn8\" (UniqueName: \"kubernetes.io/projected/6930cf41-976c-4b06-a522-9fc7f633e8e0-kube-api-access-6zxn8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.892776 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.892725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.892776 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.892739    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.893060 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.893029    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.893060 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.893057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.893232 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.893141    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6930cf41-976c-4b06-a522-9fc7f633e8e0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.893232 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.893177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.894742 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.894722    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6930cf41-976c-4b06-a522-9fc7f633e8e0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.895128 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.895110    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6930cf41-976c-4b06-a522-9fc7f633e8e0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.900753 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.900730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6930cf41-976c-4b06-a522-9fc7f633e8e0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:28.901000 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:28.900979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxn8\" (UniqueName: \"kubernetes.io/projected/6930cf41-976c-4b06-a522-9fc7f633e8e0-kube-api-access-6zxn8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p\" (UID: \"6930cf41-976c-4b06-a522-9fc7f633e8e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:29.042666 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:29.042581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:29.319633 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:29.319505    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"]
Sep 15 11:45:29.322088 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:45:29.322058    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6930cf41_976c_4b06_a522_9fc7f633e8e0.slice/crio-462b4696dd70758139fcd1ce9c883fdccf4649d6050220c9c82638561908e8e4 WatchSource:0}: Error finding container 462b4696dd70758139fcd1ce9c883fdccf4649d6050220c9c82638561908e8e4: Status 404 returned error can't find the container with id 462b4696dd70758139fcd1ce9c883fdccf4649d6050220c9c82638561908e8e4
Sep 15 11:45:29.527592 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:29.527558    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p" event={"ID":"6930cf41-976c-4b06-a522-9fc7f633e8e0","Type":"ContainerStarted","Data":"462b4696dd70758139fcd1ce9c883fdccf4649d6050220c9c82638561908e8e4"}
Sep 15 11:45:31.834751 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:31.834717    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74d9fc7f7-65kmw"]
Sep 15 11:45:31.838535 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:31.838513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:31.852707 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:31.852681    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d9fc7f7-65kmw"]
Sep 15 11:45:31.915047 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:31.915003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e387e513-3eda-48c5-9ef8-dd565a73b8aa-service-ca\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:31.915047 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:31.915047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e387e513-3eda-48c5-9ef8-dd565a73b8aa-oauth-serving-cert\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:31.915376 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:31.915099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e387e513-3eda-48c5-9ef8-dd565a73b8aa-console-serving-cert\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:31.915376 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:31.915133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e387e513-3eda-48c5-9ef8-dd565a73b8aa-console-oauth-config\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:31.915376 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:31.915167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e387e513-3eda-48c5-9ef8-dd565a73b8aa-console-config\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:31.915376 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:31.915227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzx2x\" (UniqueName: \"kubernetes.io/projected/e387e513-3eda-48c5-9ef8-dd565a73b8aa-kube-api-access-vzx2x\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:31.915376 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:31.915329    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e387e513-3eda-48c5-9ef8-dd565a73b8aa-trusted-ca-bundle\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.015867 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.015832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzx2x\" (UniqueName: \"kubernetes.io/projected/e387e513-3eda-48c5-9ef8-dd565a73b8aa-kube-api-access-vzx2x\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.016050 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.015940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e387e513-3eda-48c5-9ef8-dd565a73b8aa-trusted-ca-bundle\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.016050 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.015981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e387e513-3eda-48c5-9ef8-dd565a73b8aa-service-ca\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.016050 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.016003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e387e513-3eda-48c5-9ef8-dd565a73b8aa-oauth-serving-cert\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.016215 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.016053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e387e513-3eda-48c5-9ef8-dd565a73b8aa-console-serving-cert\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.016215 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.016077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e387e513-3eda-48c5-9ef8-dd565a73b8aa-console-oauth-config\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.016215 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.016102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e387e513-3eda-48c5-9ef8-dd565a73b8aa-console-config\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.016733 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.016702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e387e513-3eda-48c5-9ef8-dd565a73b8aa-service-ca\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.016917 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.016861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e387e513-3eda-48c5-9ef8-dd565a73b8aa-trusted-ca-bundle\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.016988 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.016952    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e387e513-3eda-48c5-9ef8-dd565a73b8aa-console-config\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.017192 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.017165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e387e513-3eda-48c5-9ef8-dd565a73b8aa-oauth-serving-cert\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.019115 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.019083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e387e513-3eda-48c5-9ef8-dd565a73b8aa-console-oauth-config\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.019300 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.019283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e387e513-3eda-48c5-9ef8-dd565a73b8aa-console-serving-cert\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.024195 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.024173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzx2x\" (UniqueName: \"kubernetes.io/projected/e387e513-3eda-48c5-9ef8-dd565a73b8aa-kube-api-access-vzx2x\") pod \"console-74d9fc7f7-65kmw\" (UID: \"e387e513-3eda-48c5-9ef8-dd565a73b8aa\") " pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.148099 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.148020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:32.876899 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.876824    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:45:32.877280 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.876939    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:45:32.877280 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:32.876986    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:45:33.075315 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.075287    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d9fc7f7-65kmw"]
Sep 15 11:45:33.078747 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:45:33.078714    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode387e513_3eda_48c5_9ef8_dd565a73b8aa.slice/crio-20516ebc36146ceb33e478e426ee84b4514a9d8d0b7bb36e9dd105b52986c13b WatchSource:0}: Error finding container 20516ebc36146ceb33e478e426ee84b4514a9d8d0b7bb36e9dd105b52986c13b: Status 404 returned error can't find the container with id 20516ebc36146ceb33e478e426ee84b4514a9d8d0b7bb36e9dd105b52986c13b
Sep 15 11:45:33.233967 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.233935    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-97ktw"]
Sep 15 11:45:33.237184 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.237162    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-97ktw"
Sep 15 11:45:33.239536 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.239512    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 15 11:45:33.239536 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.239532    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-k6k9x\""
Sep 15 11:45:33.248249 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.248225    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-97ktw"]
Sep 15 11:45:33.327417 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.327340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36595136-4851-48d3-9beb-28b17fda498c-cert\") pod \"odh-model-controller-7998c67dc4-97ktw\" (UID: \"36595136-4851-48d3-9beb-28b17fda498c\") " pod="opendatahub/odh-model-controller-7998c67dc4-97ktw"
Sep 15 11:45:33.327417 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.327377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2ngs\" (UniqueName: \"kubernetes.io/projected/36595136-4851-48d3-9beb-28b17fda498c-kube-api-access-d2ngs\") pod \"odh-model-controller-7998c67dc4-97ktw\" (UID: \"36595136-4851-48d3-9beb-28b17fda498c\") " pod="opendatahub/odh-model-controller-7998c67dc4-97ktw"
Sep 15 11:45:33.428627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.428590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36595136-4851-48d3-9beb-28b17fda498c-cert\") pod \"odh-model-controller-7998c67dc4-97ktw\" (UID: \"36595136-4851-48d3-9beb-28b17fda498c\") " pod="opendatahub/odh-model-controller-7998c67dc4-97ktw"
Sep 15 11:45:33.428804 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.428639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2ngs\" (UniqueName: \"kubernetes.io/projected/36595136-4851-48d3-9beb-28b17fda498c-kube-api-access-d2ngs\") pod \"odh-model-controller-7998c67dc4-97ktw\" (UID: \"36595136-4851-48d3-9beb-28b17fda498c\") " pod="opendatahub/odh-model-controller-7998c67dc4-97ktw"
Sep 15 11:45:33.428804 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:45:33.428742    2580 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 15 11:45:33.428804 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:45:33.428802    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36595136-4851-48d3-9beb-28b17fda498c-cert podName:36595136-4851-48d3-9beb-28b17fda498c nodeName:}" failed. No retries permitted until 2026-09-15 11:45:33.928787522 +0000 UTC m=+422.709358888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/36595136-4851-48d3-9beb-28b17fda498c-cert") pod "odh-model-controller-7998c67dc4-97ktw" (UID: "36595136-4851-48d3-9beb-28b17fda498c") : secret "odh-model-controller-webhook-cert" not found
Sep 15 11:45:33.437452 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.437423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2ngs\" (UniqueName: \"kubernetes.io/projected/36595136-4851-48d3-9beb-28b17fda498c-kube-api-access-d2ngs\") pod \"odh-model-controller-7998c67dc4-97ktw\" (UID: \"36595136-4851-48d3-9beb-28b17fda498c\") " pod="opendatahub/odh-model-controller-7998c67dc4-97ktw"
Sep 15 11:45:33.547463 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.547423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p" event={"ID":"6930cf41-976c-4b06-a522-9fc7f633e8e0","Type":"ContainerStarted","Data":"5c6a311b04d46fd447ae9528fa8d8a530e63a31663b3472319e0d5dfc54dfd76"}
Sep 15 11:45:33.549362 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.549335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d9fc7f7-65kmw" event={"ID":"e387e513-3eda-48c5-9ef8-dd565a73b8aa","Type":"ContainerStarted","Data":"b0441a1a1f5ad7ca1a46ec9d8f402ad6f9ff0519bb769eabfb6897d6af8be21c"}
Sep 15 11:45:33.549503 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.549373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d9fc7f7-65kmw" event={"ID":"e387e513-3eda-48c5-9ef8-dd565a73b8aa","Type":"ContainerStarted","Data":"20516ebc36146ceb33e478e426ee84b4514a9d8d0b7bb36e9dd105b52986c13b"}
Sep 15 11:45:33.567732 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.567676    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p" podStartSLOduration=2.015707285 podStartE2EDuration="5.567656075s" totalImagesPullingTime="3.55194879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:33.566271528 +0000 UTC m=+422.346842916" watchObservedRunningTime="2026-09-15 11:45:33.567656075 +0000 UTC m=+422.348227465"
Sep 15 11:45:33.584843 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.584750    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74d9fc7f7-65kmw" podStartSLOduration=2.5847371839999997 podStartE2EDuration="2.584737184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:33.583080586 +0000 UTC m=+422.363651974" watchObservedRunningTime="2026-09-15 11:45:33.584737184 +0000 UTC m=+422.365308572"
Sep 15 11:45:33.821334 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.821292    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mxqg4"]
Sep 15 11:45:33.825074 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.825054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mxqg4"
Sep 15 11:45:33.827341 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.827317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 15 11:45:33.827468 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.827324    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-lz7kb\""
Sep 15 11:45:33.832008 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.831846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7008f68e-d6ba-4717-a8da-9976436b8b89-tls-certs\") pod \"model-serving-api-54d89bd5fc-mxqg4\" (UID: \"7008f68e-d6ba-4717-a8da-9976436b8b89\") " pod="opendatahub/model-serving-api-54d89bd5fc-mxqg4"
Sep 15 11:45:33.832008 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.831895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25x56\" (UniqueName: \"kubernetes.io/projected/7008f68e-d6ba-4717-a8da-9976436b8b89-kube-api-access-25x56\") pod \"model-serving-api-54d89bd5fc-mxqg4\" (UID: \"7008f68e-d6ba-4717-a8da-9976436b8b89\") " pod="opendatahub/model-serving-api-54d89bd5fc-mxqg4"
Sep 15 11:45:33.834670 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.834647    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mxqg4"]
Sep 15 11:45:33.932960 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.932920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7008f68e-d6ba-4717-a8da-9976436b8b89-tls-certs\") pod \"model-serving-api-54d89bd5fc-mxqg4\" (UID: \"7008f68e-d6ba-4717-a8da-9976436b8b89\") " pod="opendatahub/model-serving-api-54d89bd5fc-mxqg4"
Sep 15 11:45:33.932960 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.932958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25x56\" (UniqueName: \"kubernetes.io/projected/7008f68e-d6ba-4717-a8da-9976436b8b89-kube-api-access-25x56\") pod \"model-serving-api-54d89bd5fc-mxqg4\" (UID: \"7008f68e-d6ba-4717-a8da-9976436b8b89\") " pod="opendatahub/model-serving-api-54d89bd5fc-mxqg4"
Sep 15 11:45:33.933501 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.932989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36595136-4851-48d3-9beb-28b17fda498c-cert\") pod \"odh-model-controller-7998c67dc4-97ktw\" (UID: \"36595136-4851-48d3-9beb-28b17fda498c\") " pod="opendatahub/odh-model-controller-7998c67dc4-97ktw"
Sep 15 11:45:33.933501 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:45:33.933075    2580 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 15 11:45:33.933501 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:45:33.933143    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7008f68e-d6ba-4717-a8da-9976436b8b89-tls-certs podName:7008f68e-d6ba-4717-a8da-9976436b8b89 nodeName:}" failed. No retries permitted until 2026-09-15 11:45:34.433124497 +0000 UTC m=+423.213695862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7008f68e-d6ba-4717-a8da-9976436b8b89-tls-certs") pod "model-serving-api-54d89bd5fc-mxqg4" (UID: "7008f68e-d6ba-4717-a8da-9976436b8b89") : secret "model-serving-api-tls" not found
Sep 15 11:45:33.935563 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.935533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36595136-4851-48d3-9beb-28b17fda498c-cert\") pod \"odh-model-controller-7998c67dc4-97ktw\" (UID: \"36595136-4851-48d3-9beb-28b17fda498c\") " pod="opendatahub/odh-model-controller-7998c67dc4-97ktw"
Sep 15 11:45:33.944028 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:33.944004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25x56\" (UniqueName: \"kubernetes.io/projected/7008f68e-d6ba-4717-a8da-9976436b8b89-kube-api-access-25x56\") pod \"model-serving-api-54d89bd5fc-mxqg4\" (UID: \"7008f68e-d6ba-4717-a8da-9976436b8b89\") " pod="opendatahub/model-serving-api-54d89bd5fc-mxqg4"
Sep 15 11:45:34.043195 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:34.043159    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:34.047986 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:34.047958    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:34.148584 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:34.148544    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-97ktw"
Sep 15 11:45:34.421296 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:34.421273    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-97ktw"]
Sep 15 11:45:34.423222 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:45:34.423200    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36595136_4851_48d3_9beb_28b17fda498c.slice/crio-098ea2d023a2ca92757d38be67698da4e587f0d8e5f68039a3e9bd36295662c1 WatchSource:0}: Error finding container 098ea2d023a2ca92757d38be67698da4e587f0d8e5f68039a3e9bd36295662c1: Status 404 returned error can't find the container with id 098ea2d023a2ca92757d38be67698da4e587f0d8e5f68039a3e9bd36295662c1
Sep 15 11:45:34.446195 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:34.446174    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7008f68e-d6ba-4717-a8da-9976436b8b89-tls-certs\") pod \"model-serving-api-54d89bd5fc-mxqg4\" (UID: \"7008f68e-d6ba-4717-a8da-9976436b8b89\") " pod="opendatahub/model-serving-api-54d89bd5fc-mxqg4"
Sep 15 11:45:34.448622 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:34.448552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7008f68e-d6ba-4717-a8da-9976436b8b89-tls-certs\") pod \"model-serving-api-54d89bd5fc-mxqg4\" (UID: \"7008f68e-d6ba-4717-a8da-9976436b8b89\") " pod="opendatahub/model-serving-api-54d89bd5fc-mxqg4"
Sep 15 11:45:34.559471 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:34.559436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-97ktw" event={"ID":"36595136-4851-48d3-9beb-28b17fda498c","Type":"ContainerStarted","Data":"098ea2d023a2ca92757d38be67698da4e587f0d8e5f68039a3e9bd36295662c1"}
Sep 15 11:45:34.559705 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:34.559685    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:34.560685 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:34.560664    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p"
Sep 15 11:45:34.737531 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:34.737454    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mxqg4"
Sep 15 11:45:35.011664 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:35.011631    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mxqg4"]
Sep 15 11:45:35.012307 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:45:35.012281    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7008f68e_d6ba_4717_a8da_9976436b8b89.slice/crio-ec213849b158b48f902a2db15aefb51c87867d23a7301c7d41f3f495ea96128d WatchSource:0}: Error finding container ec213849b158b48f902a2db15aefb51c87867d23a7301c7d41f3f495ea96128d: Status 404 returned error can't find the container with id ec213849b158b48f902a2db15aefb51c87867d23a7301c7d41f3f495ea96128d
Sep 15 11:45:35.568586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:35.568540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mxqg4" event={"ID":"7008f68e-d6ba-4717-a8da-9976436b8b89","Type":"ContainerStarted","Data":"ec213849b158b48f902a2db15aefb51c87867d23a7301c7d41f3f495ea96128d"}
Sep 15 11:45:38.588729 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:38.588688    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-97ktw" event={"ID":"36595136-4851-48d3-9beb-28b17fda498c","Type":"ContainerStarted","Data":"ac6a8f69fd2941bbe50e4a23a6c0adf648e69ce7d80966496d697402058431b4"}
Sep 15 11:45:38.589272 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:38.588781    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-97ktw"
Sep 15 11:45:38.590445 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:38.590420    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mxqg4" event={"ID":"7008f68e-d6ba-4717-a8da-9976436b8b89","Type":"ContainerStarted","Data":"13970a4103a7d1adb5d91c7df723726e719f1c88379c92aee7e0e4d06d0e92d0"}
Sep 15 11:45:38.590587 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:38.590568    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-mxqg4"
Sep 15 11:45:38.607126 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:38.607085    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-97ktw" podStartSLOduration=2.36680207 podStartE2EDuration="5.607071709s" totalImagesPullingTime="3.240269639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:38.605326078 +0000 UTC m=+427.385897468" watchObservedRunningTime="2026-09-15 11:45:38.607071709 +0000 UTC m=+427.387643096"
Sep 15 11:45:38.621850 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:38.621802    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-mxqg4" podStartSLOduration=2.974138682 podStartE2EDuration="5.621787855s" totalImagesPullingTime="2.647649173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:38.621381177 +0000 UTC m=+427.401952567" watchObservedRunningTime="2026-09-15 11:45:38.621787855 +0000 UTC m=+427.402359243"
Sep 15 11:45:41.210562 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.210524    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"]
Sep 15 11:45:41.213920 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.213896    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:41.216574 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.216549    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 11:45:41.216773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.216744    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 11:45:41.216897 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.216760    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rhc28\""
Sep 15 11:45:41.216992 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.216973    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:45:41.217465 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.217447    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 11:45:41.217593 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.217485    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 11:45:41.223911 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.223892    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"]
Sep 15 11:45:41.308070 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.308042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g5ss\" (UniqueName: \"kubernetes.io/projected/8043bd57-fec0-49a0-8889-fe42a0a9c253-kube-api-access-8g5ss\") pod \"lws-controller-manager-c47996ddf-ppd9d\" (UID: \"8043bd57-fec0-49a0-8889-fe42a0a9c253\") " pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:41.308218 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.308112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8043bd57-fec0-49a0-8889-fe42a0a9c253-manager-config\") pod \"lws-controller-manager-c47996ddf-ppd9d\" (UID: \"8043bd57-fec0-49a0-8889-fe42a0a9c253\") " pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:41.308218 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.308131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8043bd57-fec0-49a0-8889-fe42a0a9c253-cert\") pod \"lws-controller-manager-c47996ddf-ppd9d\" (UID: \"8043bd57-fec0-49a0-8889-fe42a0a9c253\") " pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:41.308218 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.308149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8043bd57-fec0-49a0-8889-fe42a0a9c253-metrics-cert\") pod \"lws-controller-manager-c47996ddf-ppd9d\" (UID: \"8043bd57-fec0-49a0-8889-fe42a0a9c253\") " pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:41.409204 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.409166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8043bd57-fec0-49a0-8889-fe42a0a9c253-manager-config\") pod \"lws-controller-manager-c47996ddf-ppd9d\" (UID: \"8043bd57-fec0-49a0-8889-fe42a0a9c253\") " pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:41.409204 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.409205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8043bd57-fec0-49a0-8889-fe42a0a9c253-cert\") pod \"lws-controller-manager-c47996ddf-ppd9d\" (UID: \"8043bd57-fec0-49a0-8889-fe42a0a9c253\") " pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:41.409437 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.409230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8043bd57-fec0-49a0-8889-fe42a0a9c253-metrics-cert\") pod \"lws-controller-manager-c47996ddf-ppd9d\" (UID: \"8043bd57-fec0-49a0-8889-fe42a0a9c253\") " pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:41.409437 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.409283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g5ss\" (UniqueName: \"kubernetes.io/projected/8043bd57-fec0-49a0-8889-fe42a0a9c253-kube-api-access-8g5ss\") pod \"lws-controller-manager-c47996ddf-ppd9d\" (UID: \"8043bd57-fec0-49a0-8889-fe42a0a9c253\") " pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:41.409823 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.409800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8043bd57-fec0-49a0-8889-fe42a0a9c253-manager-config\") pod \"lws-controller-manager-c47996ddf-ppd9d\" (UID: \"8043bd57-fec0-49a0-8889-fe42a0a9c253\") " pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:41.411887 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.411852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8043bd57-fec0-49a0-8889-fe42a0a9c253-metrics-cert\") pod \"lws-controller-manager-c47996ddf-ppd9d\" (UID: \"8043bd57-fec0-49a0-8889-fe42a0a9c253\") " pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:41.411959 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.411888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8043bd57-fec0-49a0-8889-fe42a0a9c253-cert\") pod \"lws-controller-manager-c47996ddf-ppd9d\" (UID: \"8043bd57-fec0-49a0-8889-fe42a0a9c253\") " pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:41.419809 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.419788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g5ss\" (UniqueName: \"kubernetes.io/projected/8043bd57-fec0-49a0-8889-fe42a0a9c253-kube-api-access-8g5ss\") pod \"lws-controller-manager-c47996ddf-ppd9d\" (UID: \"8043bd57-fec0-49a0-8889-fe42a0a9c253\") " pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:41.526768 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.526701    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:41.803654 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:41.803632    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"]
Sep 15 11:45:41.805424 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:45:41.805395    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8043bd57_fec0_49a0_8889_fe42a0a9c253.slice/crio-394597d301a25b1abf2f92108cb4a05fcfd45f4fb9054555a5f739c64490192a WatchSource:0}: Error finding container 394597d301a25b1abf2f92108cb4a05fcfd45f4fb9054555a5f739c64490192a: Status 404 returned error can't find the container with id 394597d301a25b1abf2f92108cb4a05fcfd45f4fb9054555a5f739c64490192a
Sep 15 11:45:42.148225 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:42.148135    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:42.148362 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:42.148235    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:42.152817 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:42.152795    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:42.610385 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:42.610349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d" event={"ID":"8043bd57-fec0-49a0-8889-fe42a0a9c253","Type":"ContainerStarted","Data":"394597d301a25b1abf2f92108cb4a05fcfd45f4fb9054555a5f739c64490192a"}
Sep 15 11:45:42.614372 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:42.614345    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74d9fc7f7-65kmw"
Sep 15 11:45:42.687662 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:42.687629    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c58cb7d7d-wx8f2"]
Sep 15 11:45:43.618444 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:43.618407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d" event={"ID":"8043bd57-fec0-49a0-8889-fe42a0a9c253","Type":"ContainerStarted","Data":"8c5b41debf1ccb786cfd14a6a5386bad2d51416e402ef158a0d20b1659c55e0c"}
Sep 15 11:45:43.618902 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:43.618482    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:45:43.652566 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:43.652517    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d" podStartSLOduration=0.94876352 podStartE2EDuration="2.652502613s" totalImagesPullingTime="1.703739093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:43.649271795 +0000 UTC m=+432.429843193" watchObservedRunningTime="2026-09-15 11:45:43.652502613 +0000 UTC m=+432.433074000"
Sep 15 11:45:49.597836 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:49.597801    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-97ktw"
Sep 15 11:45:49.600026 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:49.600009    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-mxqg4"
Sep 15 11:45:55.629797 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:45:55.629715    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-c47996ddf-ppd9d"
Sep 15 11:46:07.711980 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:07.711938    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c58cb7d7d-wx8f2" podUID="946d2972-ab99-4f44-a10a-16c418cfd80c" containerName="console" containerID="cri-o://624e08e1ee5b6029004ea61ca37b5c7cab63364df669a9a226c347992604b9e9" gracePeriod=15
Sep 15 11:46:07.948043 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:07.948015    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:46:08.044913 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.044819    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/946d2972-ab99-4f44-a10a-16c418cfd80c-console-oauth-config\") pod \"946d2972-ab99-4f44-a10a-16c418cfd80c\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") "
Sep 15 11:46:08.044913 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.044858    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-service-ca\") pod \"946d2972-ab99-4f44-a10a-16c418cfd80c\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") "
Sep 15 11:46:08.044913 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.044905    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-oauth-serving-cert\") pod \"946d2972-ab99-4f44-a10a-16c418cfd80c\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") "
Sep 15 11:46:08.045142 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.044942    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-trusted-ca-bundle\") pod \"946d2972-ab99-4f44-a10a-16c418cfd80c\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") "
Sep 15 11:46:08.045142 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.044983    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd9gc\" (UniqueName: \"kubernetes.io/projected/946d2972-ab99-4f44-a10a-16c418cfd80c-kube-api-access-rd9gc\") pod \"946d2972-ab99-4f44-a10a-16c418cfd80c\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") "
Sep 15 11:46:08.045241 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.045132    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-console-config\") pod \"946d2972-ab99-4f44-a10a-16c418cfd80c\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") "
Sep 15 11:46:08.045241 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.045183    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/946d2972-ab99-4f44-a10a-16c418cfd80c-console-serving-cert\") pod \"946d2972-ab99-4f44-a10a-16c418cfd80c\" (UID: \"946d2972-ab99-4f44-a10a-16c418cfd80c\") "
Sep 15 11:46:08.045424 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.045397    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "946d2972-ab99-4f44-a10a-16c418cfd80c" (UID: "946d2972-ab99-4f44-a10a-16c418cfd80c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:46:08.045484 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.045461    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-service-ca" (OuterVolumeSpecName: "service-ca") pod "946d2972-ab99-4f44-a10a-16c418cfd80c" (UID: "946d2972-ab99-4f44-a10a-16c418cfd80c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:46:08.045523 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.045478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "946d2972-ab99-4f44-a10a-16c418cfd80c" (UID: "946d2972-ab99-4f44-a10a-16c418cfd80c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:46:08.045568 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.045548    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-console-config" (OuterVolumeSpecName: "console-config") pod "946d2972-ab99-4f44-a10a-16c418cfd80c" (UID: "946d2972-ab99-4f44-a10a-16c418cfd80c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:46:08.047077 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.047050    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946d2972-ab99-4f44-a10a-16c418cfd80c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "946d2972-ab99-4f44-a10a-16c418cfd80c" (UID: "946d2972-ab99-4f44-a10a-16c418cfd80c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:46:08.047185 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.047101    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946d2972-ab99-4f44-a10a-16c418cfd80c-kube-api-access-rd9gc" (OuterVolumeSpecName: "kube-api-access-rd9gc") pod "946d2972-ab99-4f44-a10a-16c418cfd80c" (UID: "946d2972-ab99-4f44-a10a-16c418cfd80c"). InnerVolumeSpecName "kube-api-access-rd9gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:46:08.047229 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.047202    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946d2972-ab99-4f44-a10a-16c418cfd80c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "946d2972-ab99-4f44-a10a-16c418cfd80c" (UID: "946d2972-ab99-4f44-a10a-16c418cfd80c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:46:08.146285 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.146249    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-console-config\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:08.146285 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.146286    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/946d2972-ab99-4f44-a10a-16c418cfd80c-console-serving-cert\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:08.146477 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.146303    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/946d2972-ab99-4f44-a10a-16c418cfd80c-console-oauth-config\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:08.146477 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.146317    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:08.146477 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.146332    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-oauth-serving-cert\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:08.146477 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.146346    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/946d2972-ab99-4f44-a10a-16c418cfd80c-trusted-ca-bundle\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:08.146477 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.146360    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd9gc\" (UniqueName: \"kubernetes.io/projected/946d2972-ab99-4f44-a10a-16c418cfd80c-kube-api-access-rd9gc\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:08.733851 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.733813    2580 generic.go:356] "Generic (PLEG): container finished" podID="946d2972-ab99-4f44-a10a-16c418cfd80c" containerID="624e08e1ee5b6029004ea61ca37b5c7cab63364df669a9a226c347992604b9e9" exitCode=0
Sep 15 11:46:08.733851 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.733854    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c58cb7d7d-wx8f2"
Sep 15 11:46:08.734351 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.733862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c58cb7d7d-wx8f2" event={"ID":"946d2972-ab99-4f44-a10a-16c418cfd80c","Type":"ContainerDied","Data":"624e08e1ee5b6029004ea61ca37b5c7cab63364df669a9a226c347992604b9e9"}
Sep 15 11:46:08.734351 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.733933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c58cb7d7d-wx8f2" event={"ID":"946d2972-ab99-4f44-a10a-16c418cfd80c","Type":"ContainerDied","Data":"3e34d5b68966cc367fd94b38a23bb76faef16151d55b572cf680589c7ec218c7"}
Sep 15 11:46:08.734351 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.733961    2580 scope.go:117] "RemoveContainer" containerID="624e08e1ee5b6029004ea61ca37b5c7cab63364df669a9a226c347992604b9e9"
Sep 15 11:46:08.749212 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.749189    2580 scope.go:117] "RemoveContainer" containerID="624e08e1ee5b6029004ea61ca37b5c7cab63364df669a9a226c347992604b9e9"
Sep 15 11:46:08.749476 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:46:08.749458    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"624e08e1ee5b6029004ea61ca37b5c7cab63364df669a9a226c347992604b9e9\": container with ID starting with 624e08e1ee5b6029004ea61ca37b5c7cab63364df669a9a226c347992604b9e9 not found: ID does not exist" containerID="624e08e1ee5b6029004ea61ca37b5c7cab63364df669a9a226c347992604b9e9"
Sep 15 11:46:08.749530 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.749482    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"624e08e1ee5b6029004ea61ca37b5c7cab63364df669a9a226c347992604b9e9"} err="failed to get container status \"624e08e1ee5b6029004ea61ca37b5c7cab63364df669a9a226c347992604b9e9\": rpc error: code = NotFound desc = could not find container \"624e08e1ee5b6029004ea61ca37b5c7cab63364df669a9a226c347992604b9e9\": container with ID starting with 624e08e1ee5b6029004ea61ca37b5c7cab63364df669a9a226c347992604b9e9 not found: ID does not exist"
Sep 15 11:46:08.759173 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.759145    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c58cb7d7d-wx8f2"]
Sep 15 11:46:08.763859 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:08.763838    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c58cb7d7d-wx8f2"]
Sep 15 11:46:09.730678 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:09.730643    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="946d2972-ab99-4f44-a10a-16c418cfd80c" path="/var/lib/kubelet/pods/946d2972-ab99-4f44-a10a-16c418cfd80c/volumes"
Sep 15 11:46:38.168698 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.168663    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"]
Sep 15 11:46:38.169076 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.168985    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="946d2972-ab99-4f44-a10a-16c418cfd80c" containerName="console"
Sep 15 11:46:38.169076 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.168997    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="946d2972-ab99-4f44-a10a-16c418cfd80c" containerName="console"
Sep 15 11:46:38.169076 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.169048    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="946d2972-ab99-4f44-a10a-16c418cfd80c" containerName="console"
Sep 15 11:46:38.173180 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.173163    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"
Sep 15 11:46:38.175602 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.175581    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 11:46:38.175717 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.175613    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 11:46:38.176293 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.176277    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bzsrv\""
Sep 15 11:46:38.181724 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.181698    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"]
Sep 15 11:46:38.267563 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.267538    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"]
Sep 15 11:46:38.270768 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.270744    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"
Sep 15 11:46:38.271040 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.270843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3521e8a7-82d9-41e7-9b96-94e63868599a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm\" (UID: \"3521e8a7-82d9-41e7-9b96-94e63868599a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"
Sep 15 11:46:38.271374 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.271351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms7vn\" (UniqueName: \"kubernetes.io/projected/3521e8a7-82d9-41e7-9b96-94e63868599a-kube-api-access-ms7vn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm\" (UID: \"3521e8a7-82d9-41e7-9b96-94e63868599a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"
Sep 15 11:46:38.271463 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.271413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3521e8a7-82d9-41e7-9b96-94e63868599a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm\" (UID: \"3521e8a7-82d9-41e7-9b96-94e63868599a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"
Sep 15 11:46:38.280982 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.280958    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"]
Sep 15 11:46:38.365316 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.365284    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"]
Sep 15 11:46:38.368597 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.368580    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"
Sep 15 11:46:38.372354 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.372328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0e73014-6d94-4474-9e63-78dab40968e1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7\" (UID: \"e0e73014-6d94-4474-9e63-78dab40968e1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"
Sep 15 11:46:38.372463 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.372375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms7vn\" (UniqueName: \"kubernetes.io/projected/3521e8a7-82d9-41e7-9b96-94e63868599a-kube-api-access-ms7vn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm\" (UID: \"3521e8a7-82d9-41e7-9b96-94e63868599a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"
Sep 15 11:46:38.372463 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.372411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3521e8a7-82d9-41e7-9b96-94e63868599a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm\" (UID: \"3521e8a7-82d9-41e7-9b96-94e63868599a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"
Sep 15 11:46:38.372463 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.372434    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0e73014-6d94-4474-9e63-78dab40968e1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7\" (UID: \"e0e73014-6d94-4474-9e63-78dab40968e1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"
Sep 15 11:46:38.372463 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.372453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp4c5\" (UniqueName: \"kubernetes.io/projected/e0e73014-6d94-4474-9e63-78dab40968e1-kube-api-access-pp4c5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7\" (UID: \"e0e73014-6d94-4474-9e63-78dab40968e1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"
Sep 15 11:46:38.372691 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.372486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3521e8a7-82d9-41e7-9b96-94e63868599a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm\" (UID: \"3521e8a7-82d9-41e7-9b96-94e63868599a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"
Sep 15 11:46:38.372776 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.372759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3521e8a7-82d9-41e7-9b96-94e63868599a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm\" (UID: \"3521e8a7-82d9-41e7-9b96-94e63868599a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"
Sep 15 11:46:38.372830 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.372765    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3521e8a7-82d9-41e7-9b96-94e63868599a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm\" (UID: \"3521e8a7-82d9-41e7-9b96-94e63868599a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"
Sep 15 11:46:38.377435 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.377410    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"]
Sep 15 11:46:38.380835 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.380812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms7vn\" (UniqueName: \"kubernetes.io/projected/3521e8a7-82d9-41e7-9b96-94e63868599a-kube-api-access-ms7vn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm\" (UID: \"3521e8a7-82d9-41e7-9b96-94e63868599a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"
Sep 15 11:46:38.473172 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.473090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0e73014-6d94-4474-9e63-78dab40968e1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7\" (UID: \"e0e73014-6d94-4474-9e63-78dab40968e1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"
Sep 15 11:46:38.473172 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.473156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61d98ef1-d9ce-4542-ab04-f039f609c41e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84\" (UID: \"61d98ef1-d9ce-4542-ab04-f039f609c41e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"
Sep 15 11:46:38.473365 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.473202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0e73014-6d94-4474-9e63-78dab40968e1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7\" (UID: \"e0e73014-6d94-4474-9e63-78dab40968e1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"
Sep 15 11:46:38.473365 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.473232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp4c5\" (UniqueName: \"kubernetes.io/projected/e0e73014-6d94-4474-9e63-78dab40968e1-kube-api-access-pp4c5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7\" (UID: \"e0e73014-6d94-4474-9e63-78dab40968e1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"
Sep 15 11:46:38.473365 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.473283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61d98ef1-d9ce-4542-ab04-f039f609c41e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84\" (UID: \"61d98ef1-d9ce-4542-ab04-f039f609c41e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"
Sep 15 11:46:38.473365 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.473309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqmj\" (UniqueName: \"kubernetes.io/projected/61d98ef1-d9ce-4542-ab04-f039f609c41e-kube-api-access-lrqmj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84\" (UID: \"61d98ef1-d9ce-4542-ab04-f039f609c41e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"
Sep 15 11:46:38.473523 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.473470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0e73014-6d94-4474-9e63-78dab40968e1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7\" (UID: \"e0e73014-6d94-4474-9e63-78dab40968e1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"
Sep 15 11:46:38.473580 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.473558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0e73014-6d94-4474-9e63-78dab40968e1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7\" (UID: \"e0e73014-6d94-4474-9e63-78dab40968e1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"
Sep 15 11:46:38.474076 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.474059    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"]
Sep 15 11:46:38.477665 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.477651    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"
Sep 15 11:46:38.482524 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.482500    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"
Sep 15 11:46:38.486039 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.486020    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"]
Sep 15 11:46:38.486966 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.486946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp4c5\" (UniqueName: \"kubernetes.io/projected/e0e73014-6d94-4474-9e63-78dab40968e1-kube-api-access-pp4c5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7\" (UID: \"e0e73014-6d94-4474-9e63-78dab40968e1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"
Sep 15 11:46:38.573770 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.573734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk64m\" (UniqueName: \"kubernetes.io/projected/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-kube-api-access-zk64m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv\" (UID: \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"
Sep 15 11:46:38.573948 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.573816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61d98ef1-d9ce-4542-ab04-f039f609c41e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84\" (UID: \"61d98ef1-d9ce-4542-ab04-f039f609c41e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"
Sep 15 11:46:38.573948 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.573844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqmj\" (UniqueName: \"kubernetes.io/projected/61d98ef1-d9ce-4542-ab04-f039f609c41e-kube-api-access-lrqmj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84\" (UID: \"61d98ef1-d9ce-4542-ab04-f039f609c41e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"
Sep 15 11:46:38.573948 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.573907    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv\" (UID: \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"
Sep 15 11:46:38.574095 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.573947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv\" (UID: \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"
Sep 15 11:46:38.574095 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.574047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61d98ef1-d9ce-4542-ab04-f039f609c41e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84\" (UID: \"61d98ef1-d9ce-4542-ab04-f039f609c41e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"
Sep 15 11:46:38.574256 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.574234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61d98ef1-d9ce-4542-ab04-f039f609c41e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84\" (UID: \"61d98ef1-d9ce-4542-ab04-f039f609c41e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"
Sep 15 11:46:38.574363 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.574346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61d98ef1-d9ce-4542-ab04-f039f609c41e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84\" (UID: \"61d98ef1-d9ce-4542-ab04-f039f609c41e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"
Sep 15 11:46:38.583214 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.583186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqmj\" (UniqueName: \"kubernetes.io/projected/61d98ef1-d9ce-4542-ab04-f039f609c41e-kube-api-access-lrqmj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84\" (UID: \"61d98ef1-d9ce-4542-ab04-f039f609c41e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"
Sep 15 11:46:38.586961 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.586942    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"
Sep 15 11:46:38.674523 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.674487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk64m\" (UniqueName: \"kubernetes.io/projected/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-kube-api-access-zk64m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv\" (UID: \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"
Sep 15 11:46:38.674659 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.674540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv\" (UID: \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"
Sep 15 11:46:38.674659 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.674571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv\" (UID: \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"
Sep 15 11:46:38.674969 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.674951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv\" (UID: \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"
Sep 15 11:46:38.675023 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.674990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv\" (UID: \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"
Sep 15 11:46:38.678378 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.678361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"
Sep 15 11:46:38.683522 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.683500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk64m\" (UniqueName: \"kubernetes.io/projected/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-kube-api-access-zk64m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv\" (UID: \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"
Sep 15 11:46:38.757273 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.757240    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"]
Sep 15 11:46:38.758802 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:46:38.758770    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3521e8a7_82d9_41e7_9b96_94e63868599a.slice/crio-e129aec9d4872aade7322782a36df4fe64157f6607048c2b7c5906a85dbce82b WatchSource:0}: Error finding container e129aec9d4872aade7322782a36df4fe64157f6607048c2b7c5906a85dbce82b: Status 404 returned error can't find the container with id e129aec9d4872aade7322782a36df4fe64157f6607048c2b7c5906a85dbce82b
Sep 15 11:46:38.805569 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.805545    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"
Sep 15 11:46:38.857892 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.857809    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"]
Sep 15 11:46:38.860837 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:46:38.860791    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e73014_6d94_4474_9e63_78dab40968e1.slice/crio-ae7e0b1278a295c8591810eaa65e0a68dec6ae5452a3ad44714533a3af088a23 WatchSource:0}: Error finding container ae7e0b1278a295c8591810eaa65e0a68dec6ae5452a3ad44714533a3af088a23: Status 404 returned error can't find the container with id ae7e0b1278a295c8591810eaa65e0a68dec6ae5452a3ad44714533a3af088a23
Sep 15 11:46:38.872765 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.872736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7" event={"ID":"e0e73014-6d94-4474-9e63-78dab40968e1","Type":"ContainerStarted","Data":"ae7e0b1278a295c8591810eaa65e0a68dec6ae5452a3ad44714533a3af088a23"}
Sep 15 11:46:38.874466 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.874349    2580 generic.go:356] "Generic (PLEG): container finished" podID="3521e8a7-82d9-41e7-9b96-94e63868599a" containerID="0db2465f082b9c65a840b80e511818be2ca13fb4d3c28c03504b090a4fe33651" exitCode=0
Sep 15 11:46:38.874466 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.874402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm" event={"ID":"3521e8a7-82d9-41e7-9b96-94e63868599a","Type":"ContainerDied","Data":"0db2465f082b9c65a840b80e511818be2ca13fb4d3c28c03504b090a4fe33651"}
Sep 15 11:46:38.874466 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.874435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm" event={"ID":"3521e8a7-82d9-41e7-9b96-94e63868599a","Type":"ContainerStarted","Data":"e129aec9d4872aade7322782a36df4fe64157f6607048c2b7c5906a85dbce82b"}
Sep 15 11:46:38.956164 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:38.956141    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"]
Sep 15 11:46:38.958768 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:46:38.958740    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d98ef1_d9ce_4542_ab04_f039f609c41e.slice/crio-1fe9b1964822846974b253fc05a2a42f63943861a3adfb5c072a7d87177d7ce6 WatchSource:0}: Error finding container 1fe9b1964822846974b253fc05a2a42f63943861a3adfb5c072a7d87177d7ce6: Status 404 returned error can't find the container with id 1fe9b1964822846974b253fc05a2a42f63943861a3adfb5c072a7d87177d7ce6
Sep 15 11:46:39.086669 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:39.086642    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"]
Sep 15 11:46:39.130663 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:46:39.130628    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96ce3df_d4ca_4a2e_acd6_b5f4fa8b2856.slice/crio-c31ef965795b8b5b70c15a25bf9d2c5f99528b9e39557ac2a5a5a8878e0e06af WatchSource:0}: Error finding container c31ef965795b8b5b70c15a25bf9d2c5f99528b9e39557ac2a5a5a8878e0e06af: Status 404 returned error can't find the container with id c31ef965795b8b5b70c15a25bf9d2c5f99528b9e39557ac2a5a5a8878e0e06af
Sep 15 11:46:39.881343 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:39.881260    2580 generic.go:356] "Generic (PLEG): container finished" podID="a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856" containerID="4ae732ef5689b41d1fd592be1f9b138488e0a85002f3ed4edc9131c6cdb71838" exitCode=0
Sep 15 11:46:39.881785 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:39.881348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv" event={"ID":"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856","Type":"ContainerDied","Data":"4ae732ef5689b41d1fd592be1f9b138488e0a85002f3ed4edc9131c6cdb71838"}
Sep 15 11:46:39.881785 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:39.881397    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv" event={"ID":"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856","Type":"ContainerStarted","Data":"c31ef965795b8b5b70c15a25bf9d2c5f99528b9e39557ac2a5a5a8878e0e06af"}
Sep 15 11:46:39.883131 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:39.883107    2580 generic.go:356] "Generic (PLEG): container finished" podID="e0e73014-6d94-4474-9e63-78dab40968e1" containerID="97859a570b6e8835cdeabf7533601b39eb4948934c09d3dfbe582fd17fdb3ce4" exitCode=0
Sep 15 11:46:39.883244 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:39.883188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7" event={"ID":"e0e73014-6d94-4474-9e63-78dab40968e1","Type":"ContainerDied","Data":"97859a570b6e8835cdeabf7533601b39eb4948934c09d3dfbe582fd17fdb3ce4"}
Sep 15 11:46:39.885303 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:39.885275    2580 generic.go:356] "Generic (PLEG): container finished" podID="3521e8a7-82d9-41e7-9b96-94e63868599a" containerID="aa3c3c4f6678d2a16c33b7013ada9039ef643d2e5fd81fee79df94042dc15d48" exitCode=0
Sep 15 11:46:39.885402 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:39.885352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm" event={"ID":"3521e8a7-82d9-41e7-9b96-94e63868599a","Type":"ContainerDied","Data":"aa3c3c4f6678d2a16c33b7013ada9039ef643d2e5fd81fee79df94042dc15d48"}
Sep 15 11:46:39.887115 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:39.887089    2580 generic.go:356] "Generic (PLEG): container finished" podID="61d98ef1-d9ce-4542-ab04-f039f609c41e" containerID="085e44d37473bef9c4fdbfa6289d1bfeda88a893d7449f5d9bc79afecdfe20a7" exitCode=0
Sep 15 11:46:39.887221 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:39.887113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84" event={"ID":"61d98ef1-d9ce-4542-ab04-f039f609c41e","Type":"ContainerDied","Data":"085e44d37473bef9c4fdbfa6289d1bfeda88a893d7449f5d9bc79afecdfe20a7"}
Sep 15 11:46:39.887221 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:39.887167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84" event={"ID":"61d98ef1-d9ce-4542-ab04-f039f609c41e","Type":"ContainerStarted","Data":"1fe9b1964822846974b253fc05a2a42f63943861a3adfb5c072a7d87177d7ce6"}
Sep 15 11:46:40.896493 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:40.896457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7" event={"ID":"e0e73014-6d94-4474-9e63-78dab40968e1","Type":"ContainerStarted","Data":"4ff253edda54fa1dfa8f370bbb2e538d3987443d9afd87322e755a2b8556ba6c"}
Sep 15 11:46:40.898856 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:40.898825    2580 generic.go:356] "Generic (PLEG): container finished" podID="3521e8a7-82d9-41e7-9b96-94e63868599a" containerID="62009cffd3e71af141eee3f272f40f9aa3d4828dc83b46f821bed76b82e9fe3a" exitCode=0
Sep 15 11:46:40.898988 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:40.898913    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm" event={"ID":"3521e8a7-82d9-41e7-9b96-94e63868599a","Type":"ContainerDied","Data":"62009cffd3e71af141eee3f272f40f9aa3d4828dc83b46f821bed76b82e9fe3a"}
Sep 15 11:46:40.901473 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:40.901435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84" event={"ID":"61d98ef1-d9ce-4542-ab04-f039f609c41e","Type":"ContainerStarted","Data":"2c89be480fda9ffb8b9284808ea706d5772aa39e2839b1c7a645e8f83a565d13"}
Sep 15 11:46:40.903896 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:40.903857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv" event={"ID":"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856","Type":"ContainerStarted","Data":"377daf37fb11053d2d4af35302b64dadb0e205d53a0038d0516d58975dae1ec9"}
Sep 15 11:46:41.910158 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:41.910128    2580 generic.go:356] "Generic (PLEG): container finished" podID="a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856" containerID="377daf37fb11053d2d4af35302b64dadb0e205d53a0038d0516d58975dae1ec9" exitCode=0
Sep 15 11:46:41.910570 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:41.910207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv" event={"ID":"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856","Type":"ContainerDied","Data":"377daf37fb11053d2d4af35302b64dadb0e205d53a0038d0516d58975dae1ec9"}
Sep 15 11:46:41.913297 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:41.913270    2580 generic.go:356] "Generic (PLEG): container finished" podID="e0e73014-6d94-4474-9e63-78dab40968e1" containerID="4ff253edda54fa1dfa8f370bbb2e538d3987443d9afd87322e755a2b8556ba6c" exitCode=0
Sep 15 11:46:41.913410 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:41.913398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7" event={"ID":"e0e73014-6d94-4474-9e63-78dab40968e1","Type":"ContainerDied","Data":"4ff253edda54fa1dfa8f370bbb2e538d3987443d9afd87322e755a2b8556ba6c"}
Sep 15 11:46:41.916065 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:41.916044    2580 generic.go:356] "Generic (PLEG): container finished" podID="61d98ef1-d9ce-4542-ab04-f039f609c41e" containerID="2c89be480fda9ffb8b9284808ea706d5772aa39e2839b1c7a645e8f83a565d13" exitCode=0
Sep 15 11:46:41.916148 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:41.916123    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84" event={"ID":"61d98ef1-d9ce-4542-ab04-f039f609c41e","Type":"ContainerDied","Data":"2c89be480fda9ffb8b9284808ea706d5772aa39e2839b1c7a645e8f83a565d13"}
Sep 15 11:46:42.068784 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.068758    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"
Sep 15 11:46:42.104442 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.104414    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3521e8a7-82d9-41e7-9b96-94e63868599a-bundle\") pod \"3521e8a7-82d9-41e7-9b96-94e63868599a\" (UID: \"3521e8a7-82d9-41e7-9b96-94e63868599a\") "
Sep 15 11:46:42.104442 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.104446    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3521e8a7-82d9-41e7-9b96-94e63868599a-util\") pod \"3521e8a7-82d9-41e7-9b96-94e63868599a\" (UID: \"3521e8a7-82d9-41e7-9b96-94e63868599a\") "
Sep 15 11:46:42.104635 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.104529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms7vn\" (UniqueName: \"kubernetes.io/projected/3521e8a7-82d9-41e7-9b96-94e63868599a-kube-api-access-ms7vn\") pod \"3521e8a7-82d9-41e7-9b96-94e63868599a\" (UID: \"3521e8a7-82d9-41e7-9b96-94e63868599a\") "
Sep 15 11:46:42.104938 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.104912    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3521e8a7-82d9-41e7-9b96-94e63868599a-bundle" (OuterVolumeSpecName: "bundle") pod "3521e8a7-82d9-41e7-9b96-94e63868599a" (UID: "3521e8a7-82d9-41e7-9b96-94e63868599a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:46:42.106515 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.106493    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3521e8a7-82d9-41e7-9b96-94e63868599a-kube-api-access-ms7vn" (OuterVolumeSpecName: "kube-api-access-ms7vn") pod "3521e8a7-82d9-41e7-9b96-94e63868599a" (UID: "3521e8a7-82d9-41e7-9b96-94e63868599a"). InnerVolumeSpecName "kube-api-access-ms7vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:46:42.110757 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.110733    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3521e8a7-82d9-41e7-9b96-94e63868599a-util" (OuterVolumeSpecName: "util") pod "3521e8a7-82d9-41e7-9b96-94e63868599a" (UID: "3521e8a7-82d9-41e7-9b96-94e63868599a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:46:42.205974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.205943    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3521e8a7-82d9-41e7-9b96-94e63868599a-bundle\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:42.205974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.205971    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3521e8a7-82d9-41e7-9b96-94e63868599a-util\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:42.206139 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.205984    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms7vn\" (UniqueName: \"kubernetes.io/projected/3521e8a7-82d9-41e7-9b96-94e63868599a-kube-api-access-ms7vn\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:42.922611 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.922575    2580 generic.go:356] "Generic (PLEG): container finished" podID="a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856" containerID="7dd9384635c087fbd538bccbe892bf80875aeeda8e211f2d292cc92879b7142f" exitCode=0
Sep 15 11:46:42.923021 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.922659    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv" event={"ID":"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856","Type":"ContainerDied","Data":"7dd9384635c087fbd538bccbe892bf80875aeeda8e211f2d292cc92879b7142f"}
Sep 15 11:46:42.924780 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.924757    2580 generic.go:356] "Generic (PLEG): container finished" podID="e0e73014-6d94-4474-9e63-78dab40968e1" containerID="8b6dd9731c7356f6d771e3c6876c123810ecb9e67f889749655e18e77764b5dc" exitCode=0
Sep 15 11:46:42.924914 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.924839    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7" event={"ID":"e0e73014-6d94-4474-9e63-78dab40968e1","Type":"ContainerDied","Data":"8b6dd9731c7356f6d771e3c6876c123810ecb9e67f889749655e18e77764b5dc"}
Sep 15 11:46:42.926609 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.926592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm" event={"ID":"3521e8a7-82d9-41e7-9b96-94e63868599a","Type":"ContainerDied","Data":"e129aec9d4872aade7322782a36df4fe64157f6607048c2b7c5906a85dbce82b"}
Sep 15 11:46:42.926686 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.926612    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ss4nm"
Sep 15 11:46:42.926686 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.926620    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e129aec9d4872aade7322782a36df4fe64157f6607048c2b7c5906a85dbce82b"
Sep 15 11:46:42.928771 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.928753    2580 generic.go:356] "Generic (PLEG): container finished" podID="61d98ef1-d9ce-4542-ab04-f039f609c41e" containerID="0fc39a01cc91279529b8d4d81ed15e0215a53ed76ca0191ba7376071bcf6383a" exitCode=0
Sep 15 11:46:42.928857 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:42.928798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84" event={"ID":"61d98ef1-d9ce-4542-ab04-f039f609c41e","Type":"ContainerDied","Data":"0fc39a01cc91279529b8d4d81ed15e0215a53ed76ca0191ba7376071bcf6383a"}
Sep 15 11:46:44.076577 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.076548    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"
Sep 15 11:46:44.120251 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.120229    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"
Sep 15 11:46:44.123462 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.123445    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"
Sep 15 11:46:44.126206 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.126190    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrqmj\" (UniqueName: \"kubernetes.io/projected/61d98ef1-d9ce-4542-ab04-f039f609c41e-kube-api-access-lrqmj\") pod \"61d98ef1-d9ce-4542-ab04-f039f609c41e\" (UID: \"61d98ef1-d9ce-4542-ab04-f039f609c41e\") "
Sep 15 11:46:44.126303 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.126287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61d98ef1-d9ce-4542-ab04-f039f609c41e-bundle\") pod \"61d98ef1-d9ce-4542-ab04-f039f609c41e\" (UID: \"61d98ef1-d9ce-4542-ab04-f039f609c41e\") "
Sep 15 11:46:44.126345 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.126327    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61d98ef1-d9ce-4542-ab04-f039f609c41e-util\") pod \"61d98ef1-d9ce-4542-ab04-f039f609c41e\" (UID: \"61d98ef1-d9ce-4542-ab04-f039f609c41e\") "
Sep 15 11:46:44.126885 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.126842    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d98ef1-d9ce-4542-ab04-f039f609c41e-bundle" (OuterVolumeSpecName: "bundle") pod "61d98ef1-d9ce-4542-ab04-f039f609c41e" (UID: "61d98ef1-d9ce-4542-ab04-f039f609c41e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:46:44.128257 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.128231    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d98ef1-d9ce-4542-ab04-f039f609c41e-kube-api-access-lrqmj" (OuterVolumeSpecName: "kube-api-access-lrqmj") pod "61d98ef1-d9ce-4542-ab04-f039f609c41e" (UID: "61d98ef1-d9ce-4542-ab04-f039f609c41e"). InnerVolumeSpecName "kube-api-access-lrqmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:46:44.131651 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.131630    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d98ef1-d9ce-4542-ab04-f039f609c41e-util" (OuterVolumeSpecName: "util") pod "61d98ef1-d9ce-4542-ab04-f039f609c41e" (UID: "61d98ef1-d9ce-4542-ab04-f039f609c41e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:46:44.227380 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.227298    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0e73014-6d94-4474-9e63-78dab40968e1-bundle\") pod \"e0e73014-6d94-4474-9e63-78dab40968e1\" (UID: \"e0e73014-6d94-4474-9e63-78dab40968e1\") "
Sep 15 11:46:44.227380 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.227347    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-util\") pod \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\" (UID: \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\") "
Sep 15 11:46:44.227586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.227399    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-bundle\") pod \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\" (UID: \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\") "
Sep 15 11:46:44.227586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.227451    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk64m\" (UniqueName: \"kubernetes.io/projected/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-kube-api-access-zk64m\") pod \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\" (UID: \"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856\") "
Sep 15 11:46:44.227586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.227496    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp4c5\" (UniqueName: \"kubernetes.io/projected/e0e73014-6d94-4474-9e63-78dab40968e1-kube-api-access-pp4c5\") pod \"e0e73014-6d94-4474-9e63-78dab40968e1\" (UID: \"e0e73014-6d94-4474-9e63-78dab40968e1\") "
Sep 15 11:46:44.227586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.227553    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0e73014-6d94-4474-9e63-78dab40968e1-util\") pod \"e0e73014-6d94-4474-9e63-78dab40968e1\" (UID: \"e0e73014-6d94-4474-9e63-78dab40968e1\") "
Sep 15 11:46:44.227816 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.227795    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61d98ef1-d9ce-4542-ab04-f039f609c41e-bundle\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:44.227897 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.227821    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61d98ef1-d9ce-4542-ab04-f039f609c41e-util\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:44.227897 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.227835    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrqmj\" (UniqueName: \"kubernetes.io/projected/61d98ef1-d9ce-4542-ab04-f039f609c41e-kube-api-access-lrqmj\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:44.228019 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.227990    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e73014-6d94-4474-9e63-78dab40968e1-bundle" (OuterVolumeSpecName: "bundle") pod "e0e73014-6d94-4474-9e63-78dab40968e1" (UID: "e0e73014-6d94-4474-9e63-78dab40968e1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:46:44.228146 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.228121    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-bundle" (OuterVolumeSpecName: "bundle") pod "a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856" (UID: "a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:46:44.229700 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.229678    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-kube-api-access-zk64m" (OuterVolumeSpecName: "kube-api-access-zk64m") pod "a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856" (UID: "a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856"). InnerVolumeSpecName "kube-api-access-zk64m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:46:44.230113 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.230091    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e73014-6d94-4474-9e63-78dab40968e1-kube-api-access-pp4c5" (OuterVolumeSpecName: "kube-api-access-pp4c5") pod "e0e73014-6d94-4474-9e63-78dab40968e1" (UID: "e0e73014-6d94-4474-9e63-78dab40968e1"). InnerVolumeSpecName "kube-api-access-pp4c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:46:44.233534 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.233506    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e73014-6d94-4474-9e63-78dab40968e1-util" (OuterVolumeSpecName: "util") pod "e0e73014-6d94-4474-9e63-78dab40968e1" (UID: "e0e73014-6d94-4474-9e63-78dab40968e1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:46:44.234117 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.234092    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-util" (OuterVolumeSpecName: "util") pod "a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856" (UID: "a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:46:44.328723 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.328696    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk64m\" (UniqueName: \"kubernetes.io/projected/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-kube-api-access-zk64m\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:44.328723 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.328718    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp4c5\" (UniqueName: \"kubernetes.io/projected/e0e73014-6d94-4474-9e63-78dab40968e1-kube-api-access-pp4c5\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:44.328723 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.328729    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0e73014-6d94-4474-9e63-78dab40968e1-util\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:44.328949 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.328739    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0e73014-6d94-4474-9e63-78dab40968e1-bundle\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:44.328949 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.328749    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-util\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:44.328949 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.328756    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856-bundle\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:46:44.940974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.940941    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7"
Sep 15 11:46:44.941148 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.940940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8gh7" event={"ID":"e0e73014-6d94-4474-9e63-78dab40968e1","Type":"ContainerDied","Data":"ae7e0b1278a295c8591810eaa65e0a68dec6ae5452a3ad44714533a3af088a23"}
Sep 15 11:46:44.941148 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.941069    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae7e0b1278a295c8591810eaa65e0a68dec6ae5452a3ad44714533a3af088a23"
Sep 15 11:46:44.942856 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.942831    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84"
Sep 15 11:46:44.942997 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.942836    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqxn84" event={"ID":"61d98ef1-d9ce-4542-ab04-f039f609c41e","Type":"ContainerDied","Data":"1fe9b1964822846974b253fc05a2a42f63943861a3adfb5c072a7d87177d7ce6"}
Sep 15 11:46:44.942997 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.942926    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fe9b1964822846974b253fc05a2a42f63943861a3adfb5c072a7d87177d7ce6"
Sep 15 11:46:44.944706 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.944687    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv" event={"ID":"a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856","Type":"ContainerDied","Data":"c31ef965795b8b5b70c15a25bf9d2c5f99528b9e39557ac2a5a5a8878e0e06af"}
Sep 15 11:46:44.944803 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.944713    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs9kmv"
Sep 15 11:46:44.944803 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:46:44.944716    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c31ef965795b8b5b70c15a25bf9d2c5f99528b9e39557ac2a5a5a8878e0e06af"
Sep 15 11:48:18.317423 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317381    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-qzt8h"]
Sep 15 11:48:18.317974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317824    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61d98ef1-d9ce-4542-ab04-f039f609c41e" containerName="pull"
Sep 15 11:48:18.317974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317843    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="61d98ef1-d9ce-4542-ab04-f039f609c41e" containerName="pull"
Sep 15 11:48:18.317974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317860    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856" containerName="extract"
Sep 15 11:48:18.317974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317886    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856" containerName="extract"
Sep 15 11:48:18.317974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317901    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856" containerName="pull"
Sep 15 11:48:18.317974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317908    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856" containerName="pull"
Sep 15 11:48:18.317974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317922    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e73014-6d94-4474-9e63-78dab40968e1" containerName="pull"
Sep 15 11:48:18.317974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317930    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e73014-6d94-4474-9e63-78dab40968e1" containerName="pull"
Sep 15 11:48:18.317974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317942    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3521e8a7-82d9-41e7-9b96-94e63868599a" containerName="extract"
Sep 15 11:48:18.317974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317950    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3521e8a7-82d9-41e7-9b96-94e63868599a" containerName="extract"
Sep 15 11:48:18.317974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317961    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3521e8a7-82d9-41e7-9b96-94e63868599a" containerName="pull"
Sep 15 11:48:18.317974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317968    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3521e8a7-82d9-41e7-9b96-94e63868599a" containerName="pull"
Sep 15 11:48:18.317974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317979    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e73014-6d94-4474-9e63-78dab40968e1" containerName="util"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317987    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e73014-6d94-4474-9e63-78dab40968e1" containerName="util"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.317996    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61d98ef1-d9ce-4542-ab04-f039f609c41e" containerName="util"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.318003    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="61d98ef1-d9ce-4542-ab04-f039f609c41e" containerName="util"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.318012    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e73014-6d94-4474-9e63-78dab40968e1" containerName="extract"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.318019    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e73014-6d94-4474-9e63-78dab40968e1" containerName="extract"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.318033    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3521e8a7-82d9-41e7-9b96-94e63868599a" containerName="util"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.318040    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3521e8a7-82d9-41e7-9b96-94e63868599a" containerName="util"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.318057    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856" containerName="util"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.318064    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856" containerName="util"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.318074    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61d98ef1-d9ce-4542-ab04-f039f609c41e" containerName="extract"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.318082    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="61d98ef1-d9ce-4542-ab04-f039f609c41e" containerName="extract"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.318172    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="61d98ef1-d9ce-4542-ab04-f039f609c41e" containerName="extract"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.318186    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3521e8a7-82d9-41e7-9b96-94e63868599a" containerName="extract"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.318195    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0e73014-6d94-4474-9e63-78dab40968e1" containerName="extract"
Sep 15 11:48:18.318586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.318204    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96ce3df-d4ca-4a2e-acd6-b5f4fa8b2856" containerName="extract"
Sep 15 11:48:18.321304 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.321284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qzt8h"
Sep 15 11:48:18.323316 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.323292    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 11:48:18.323418 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.323354    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 11:48:18.324132 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.324112    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5lr4b\""
Sep 15 11:48:18.330599 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.330568    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qzt8h"]
Sep 15 11:48:18.506450 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.506414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg97p\" (UniqueName: \"kubernetes.io/projected/0be1d97d-e7f5-4ffb-b5ac-5190168db88b-kube-api-access-vg97p\") pod \"authorino-59c974bd48-qzt8h\" (UID: \"0be1d97d-e7f5-4ffb-b5ac-5190168db88b\") " pod="kuadrant-system/authorino-59c974bd48-qzt8h"
Sep 15 11:48:18.607058 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.606976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg97p\" (UniqueName: \"kubernetes.io/projected/0be1d97d-e7f5-4ffb-b5ac-5190168db88b-kube-api-access-vg97p\") pod \"authorino-59c974bd48-qzt8h\" (UID: \"0be1d97d-e7f5-4ffb-b5ac-5190168db88b\") " pod="kuadrant-system/authorino-59c974bd48-qzt8h"
Sep 15 11:48:18.615657 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.615627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg97p\" (UniqueName: \"kubernetes.io/projected/0be1d97d-e7f5-4ffb-b5ac-5190168db88b-kube-api-access-vg97p\") pod \"authorino-59c974bd48-qzt8h\" (UID: \"0be1d97d-e7f5-4ffb-b5ac-5190168db88b\") " pod="kuadrant-system/authorino-59c974bd48-qzt8h"
Sep 15 11:48:18.638509 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.638481    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qzt8h"
Sep 15 11:48:18.906752 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:18.906726    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qzt8h"]
Sep 15 11:48:18.908802 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:48:18.908775    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0be1d97d_e7f5_4ffb_b5ac_5190168db88b.slice/crio-7c40cab6df011cfcfa4e5ea931102012568d78a0e1485d29aa71327988003844 WatchSource:0}: Error finding container 7c40cab6df011cfcfa4e5ea931102012568d78a0e1485d29aa71327988003844: Status 404 returned error can't find the container with id 7c40cab6df011cfcfa4e5ea931102012568d78a0e1485d29aa71327988003844
Sep 15 11:48:19.410849 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:19.410806    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qzt8h" event={"ID":"0be1d97d-e7f5-4ffb-b5ac-5190168db88b","Type":"ContainerStarted","Data":"7c40cab6df011cfcfa4e5ea931102012568d78a0e1485d29aa71327988003844"}
Sep 15 11:48:22.429340 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:22.429305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qzt8h" event={"ID":"0be1d97d-e7f5-4ffb-b5ac-5190168db88b","Type":"ContainerStarted","Data":"ac1d2c233c85d13fd158b1297d98458d8ea7cb0867a9c5b8ef4bf7823df0dd7d"}
Sep 15 11:48:22.448549 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:22.448497    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-qzt8h" podStartSLOduration=1.818286476 podStartE2EDuration="4.448479289s" totalImagesPullingTime="2.630192813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:48:22.4482009 +0000 UTC m=+591.228772302" watchObservedRunningTime="2026-09-15 11:48:22.448479289 +0000 UTC m=+591.229050676"
Sep 15 11:48:44.670105 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:44.670070    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qzt8h"]
Sep 15 11:48:44.670661 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:44.670250    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-qzt8h" podUID="0be1d97d-e7f5-4ffb-b5ac-5190168db88b" containerName="authorino" containerID="cri-o://ac1d2c233c85d13fd158b1297d98458d8ea7cb0867a9c5b8ef4bf7823df0dd7d" gracePeriod=30
Sep 15 11:48:44.911112 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:44.911088    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qzt8h"
Sep 15 11:48:45.022532 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:45.022439    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg97p\" (UniqueName: \"kubernetes.io/projected/0be1d97d-e7f5-4ffb-b5ac-5190168db88b-kube-api-access-vg97p\") pod \"0be1d97d-e7f5-4ffb-b5ac-5190168db88b\" (UID: \"0be1d97d-e7f5-4ffb-b5ac-5190168db88b\") "
Sep 15 11:48:45.024687 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:45.024656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be1d97d-e7f5-4ffb-b5ac-5190168db88b-kube-api-access-vg97p" (OuterVolumeSpecName: "kube-api-access-vg97p") pod "0be1d97d-e7f5-4ffb-b5ac-5190168db88b" (UID: "0be1d97d-e7f5-4ffb-b5ac-5190168db88b"). InnerVolumeSpecName "kube-api-access-vg97p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:48:45.123010 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:45.122975    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg97p\" (UniqueName: \"kubernetes.io/projected/0be1d97d-e7f5-4ffb-b5ac-5190168db88b-kube-api-access-vg97p\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:48:45.539188 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:45.539154    2580 generic.go:356] "Generic (PLEG): container finished" podID="0be1d97d-e7f5-4ffb-b5ac-5190168db88b" containerID="ac1d2c233c85d13fd158b1297d98458d8ea7cb0867a9c5b8ef4bf7823df0dd7d" exitCode=0
Sep 15 11:48:45.539359 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:45.539199    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qzt8h"
Sep 15 11:48:45.539359 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:45.539207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qzt8h" event={"ID":"0be1d97d-e7f5-4ffb-b5ac-5190168db88b","Type":"ContainerDied","Data":"ac1d2c233c85d13fd158b1297d98458d8ea7cb0867a9c5b8ef4bf7823df0dd7d"}
Sep 15 11:48:45.539359 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:45.539238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qzt8h" event={"ID":"0be1d97d-e7f5-4ffb-b5ac-5190168db88b","Type":"ContainerDied","Data":"7c40cab6df011cfcfa4e5ea931102012568d78a0e1485d29aa71327988003844"}
Sep 15 11:48:45.539359 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:45.539255    2580 scope.go:117] "RemoveContainer" containerID="ac1d2c233c85d13fd158b1297d98458d8ea7cb0867a9c5b8ef4bf7823df0dd7d"
Sep 15 11:48:45.547950 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:45.547930    2580 scope.go:117] "RemoveContainer" containerID="ac1d2c233c85d13fd158b1297d98458d8ea7cb0867a9c5b8ef4bf7823df0dd7d"
Sep 15 11:48:45.548226 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:48:45.548210    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1d2c233c85d13fd158b1297d98458d8ea7cb0867a9c5b8ef4bf7823df0dd7d\": container with ID starting with ac1d2c233c85d13fd158b1297d98458d8ea7cb0867a9c5b8ef4bf7823df0dd7d not found: ID does not exist" containerID="ac1d2c233c85d13fd158b1297d98458d8ea7cb0867a9c5b8ef4bf7823df0dd7d"
Sep 15 11:48:45.548275 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:45.548235    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1d2c233c85d13fd158b1297d98458d8ea7cb0867a9c5b8ef4bf7823df0dd7d"} err="failed to get container status \"ac1d2c233c85d13fd158b1297d98458d8ea7cb0867a9c5b8ef4bf7823df0dd7d\": rpc error: code = NotFound desc = could not find container \"ac1d2c233c85d13fd158b1297d98458d8ea7cb0867a9c5b8ef4bf7823df0dd7d\": container with ID starting with ac1d2c233c85d13fd158b1297d98458d8ea7cb0867a9c5b8ef4bf7823df0dd7d not found: ID does not exist"
Sep 15 11:48:45.561629 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:45.561598    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qzt8h"]
Sep 15 11:48:45.565975 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:45.565951    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qzt8h"]
Sep 15 11:48:45.730053 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:48:45.730023    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0be1d97d-e7f5-4ffb-b5ac-5190168db88b" path="/var/lib/kubelet/pods/0be1d97d-e7f5-4ffb-b5ac-5190168db88b/volumes"
Sep 15 11:49:27.885808 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:27.885769    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-j556w"]
Sep 15 11:49:27.886273 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:27.886081    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be1d97d-e7f5-4ffb-b5ac-5190168db88b" containerName="authorino"
Sep 15 11:49:27.886273 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:27.886092    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be1d97d-e7f5-4ffb-b5ac-5190168db88b" containerName="authorino"
Sep 15 11:49:27.886273 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:27.886153    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0be1d97d-e7f5-4ffb-b5ac-5190168db88b" containerName="authorino"
Sep 15 11:49:27.888979 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:27.888963    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-j556w"
Sep 15 11:49:27.899528 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:27.899504    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 15 11:49:27.978851 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:27.978809    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-j556w"]
Sep 15 11:49:27.981671 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:27.981642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9653379-1bc2-4fb8-9c12-9a92ca8eb998-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-j556w\" (UID: \"b9653379-1bc2-4fb8-9c12-9a92ca8eb998\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-j556w"
Sep 15 11:49:27.981782 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:27.981682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp5gm\" (UniqueName: \"kubernetes.io/projected/b9653379-1bc2-4fb8-9c12-9a92ca8eb998-kube-api-access-sp5gm\") pod \"payload-processing-5b54b7fbb7-j556w\" (UID: \"b9653379-1bc2-4fb8-9c12-9a92ca8eb998\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-j556w"
Sep 15 11:49:28.082337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:28.082298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9653379-1bc2-4fb8-9c12-9a92ca8eb998-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-j556w\" (UID: \"b9653379-1bc2-4fb8-9c12-9a92ca8eb998\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-j556w"
Sep 15 11:49:28.082337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:28.082338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp5gm\" (UniqueName: \"kubernetes.io/projected/b9653379-1bc2-4fb8-9c12-9a92ca8eb998-kube-api-access-sp5gm\") pod \"payload-processing-5b54b7fbb7-j556w\" (UID: \"b9653379-1bc2-4fb8-9c12-9a92ca8eb998\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-j556w"
Sep 15 11:49:28.083076 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:28.083055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9653379-1bc2-4fb8-9c12-9a92ca8eb998-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-j556w\" (UID: \"b9653379-1bc2-4fb8-9c12-9a92ca8eb998\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-j556w"
Sep 15 11:49:28.096815 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:28.096791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp5gm\" (UniqueName: \"kubernetes.io/projected/b9653379-1bc2-4fb8-9c12-9a92ca8eb998-kube-api-access-sp5gm\") pod \"payload-processing-5b54b7fbb7-j556w\" (UID: \"b9653379-1bc2-4fb8-9c12-9a92ca8eb998\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-j556w"
Sep 15 11:49:28.198717 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:28.198682    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-j556w"
Sep 15 11:49:28.478773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:28.478748    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-j556w"]
Sep 15 11:49:28.480497 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:49:28.480466    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9653379_1bc2_4fb8_9c12_9a92ca8eb998.slice/crio-63af3e4cad6f183840ffe385dbf7604c905cb75c11a6056dac7929b8a488d6c9 WatchSource:0}: Error finding container 63af3e4cad6f183840ffe385dbf7604c905cb75c11a6056dac7929b8a488d6c9: Status 404 returned error can't find the container with id 63af3e4cad6f183840ffe385dbf7604c905cb75c11a6056dac7929b8a488d6c9
Sep 15 11:49:28.482416 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:28.482401    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 11:49:28.752410 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:28.752333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-j556w" event={"ID":"b9653379-1bc2-4fb8-9c12-9a92ca8eb998","Type":"ContainerStarted","Data":"63af3e4cad6f183840ffe385dbf7604c905cb75c11a6056dac7929b8a488d6c9"}
Sep 15 11:49:29.758939 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:29.758903    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-j556w" event={"ID":"b9653379-1bc2-4fb8-9c12-9a92ca8eb998","Type":"ContainerStarted","Data":"e3edcbfa6ff58dd8dde7bd1d86ff046fd07aa578ddf3c32c45ba731022771699"}
Sep 15 11:49:29.759330 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:29.759049    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-j556w"
Sep 15 11:49:29.781025 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:29.780958    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-j556w" podStartSLOduration=1.587993769 podStartE2EDuration="2.780938999s" totalImagesPullingTime="1.19294523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:49:29.778308455 +0000 UTC m=+658.558879844" watchObservedRunningTime="2026-09-15 11:49:29.780938999 +0000 UTC m=+658.561510388"
Sep 15 11:49:40.767050 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:49:40.767019    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-j556w"
Sep 15 11:50:16.167365 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.167331    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"]
Sep 15 11:50:16.170703 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.170680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.173302 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.173278    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 11:50:16.174017 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.173989    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 11:50:16.174117 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.174041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 15 11:50:16.174117 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.174059    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wkm2k\""
Sep 15 11:50:16.182449 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.182431    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"]
Sep 15 11:50:16.271669 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.271638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj97w\" (UniqueName: \"kubernetes.io/projected/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-kube-api-access-pj97w\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.271846 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.271686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.271846 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.271766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.271846 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.271825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.271974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.271861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.271974 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.271910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.373037 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.373008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.373037 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.373040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.373268 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.373065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj97w\" (UniqueName: \"kubernetes.io/projected/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-kube-api-access-pj97w\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.373268 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.373114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.373268 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.373177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.373268 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.373221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.373474 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.373419    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.373533 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.373478    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.373629 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.373609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.375517 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.375484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.375630 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.375605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.382602 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.382582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj97w\" (UniqueName: \"kubernetes.io/projected/5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c-kube-api-access-pj97w\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r\" (UID: \"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.482420 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.482343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:16.756231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.756204    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"]
Sep 15 11:50:16.758311 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:50:16.758282    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2ff8be_58a6_4c6a_bd64_41cfe26ae38c.slice/crio-d7d294db90178ed50edb304b2d965b683920b2f608a8df2d82e8f1972da0ebd5 WatchSource:0}: Error finding container d7d294db90178ed50edb304b2d965b683920b2f608a8df2d82e8f1972da0ebd5: Status 404 returned error can't find the container with id d7d294db90178ed50edb304b2d965b683920b2f608a8df2d82e8f1972da0ebd5
Sep 15 11:50:16.988572 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:16.988534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r" event={"ID":"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c","Type":"ContainerStarted","Data":"d7d294db90178ed50edb304b2d965b683920b2f608a8df2d82e8f1972da0ebd5"}
Sep 15 11:50:17.995223 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:17.995127    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r" event={"ID":"5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c","Type":"ContainerStarted","Data":"7b46ef01b34be1546420693bbe70e2472ac35c54f9dc1888728fbce8e626347c"}
Sep 15 11:50:18.015035 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:18.014980    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r" podStartSLOduration=1.042649492 podStartE2EDuration="2.014964285s" totalImagesPullingTime="972.314793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:50:18.01325071 +0000 UTC m=+706.793822098" watchObservedRunningTime="2026-09-15 11:50:18.014964285 +0000 UTC m=+706.795535673"
Sep 15 11:50:26.483000 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:26.482902    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:26.483000 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:26.482960    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:26.495642 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:26.495614    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:27.053167 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:27.053132    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r"
Sep 15 11:50:54.667147 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.667115    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"]
Sep 15 11:50:54.671668 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.671651    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.673829 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.673806    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 15 11:50:54.680953 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.680929    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"]
Sep 15 11:50:54.783417 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.783384    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0a5e0c0-71ba-43aa-8272-91158e33853e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.783572 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.783441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0a5e0c0-71ba-43aa-8272-91158e33853e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.783572 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.783485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0a5e0c0-71ba-43aa-8272-91158e33853e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.783572 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.783505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh86z\" (UniqueName: \"kubernetes.io/projected/f0a5e0c0-71ba-43aa-8272-91158e33853e-kube-api-access-xh86z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.783572 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.783534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0a5e0c0-71ba-43aa-8272-91158e33853e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.783572 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.783560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0a5e0c0-71ba-43aa-8272-91158e33853e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.884850 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.884820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0a5e0c0-71ba-43aa-8272-91158e33853e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.885020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.884896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0a5e0c0-71ba-43aa-8272-91158e33853e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.885020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.884927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0a5e0c0-71ba-43aa-8272-91158e33853e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.885020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.884952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh86z\" (UniqueName: \"kubernetes.io/projected/f0a5e0c0-71ba-43aa-8272-91158e33853e-kube-api-access-xh86z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.885020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.884981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0a5e0c0-71ba-43aa-8272-91158e33853e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.885020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.885017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0a5e0c0-71ba-43aa-8272-91158e33853e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.885294 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.885273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0a5e0c0-71ba-43aa-8272-91158e33853e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.885359 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.885317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0a5e0c0-71ba-43aa-8272-91158e33853e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.885359 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.885340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0a5e0c0-71ba-43aa-8272-91158e33853e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.887233 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.887210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0a5e0c0-71ba-43aa-8272-91158e33853e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.887475 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.887460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0a5e0c0-71ba-43aa-8272-91158e33853e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.894158 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.894133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh86z\" (UniqueName: \"kubernetes.io/projected/f0a5e0c0-71ba-43aa-8272-91158e33853e-kube-api-access-xh86z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tvszs\" (UID: \"f0a5e0c0-71ba-43aa-8272-91158e33853e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:54.981759 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:54.981703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:50:55.255833 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:55.255806    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"]
Sep 15 11:50:55.257385 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:50:55.257357    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a5e0c0_71ba_43aa_8272_91158e33853e.slice/crio-4f1aec8118f7b1a732bd1f2ba0f84fd8d76328102159204abd35571365a1d63b WatchSource:0}: Error finding container 4f1aec8118f7b1a732bd1f2ba0f84fd8d76328102159204abd35571365a1d63b: Status 404 returned error can't find the container with id 4f1aec8118f7b1a732bd1f2ba0f84fd8d76328102159204abd35571365a1d63b
Sep 15 11:50:56.182760 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:56.182722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs" event={"ID":"f0a5e0c0-71ba-43aa-8272-91158e33853e","Type":"ContainerStarted","Data":"80568ceb64ec1ac09156e30cb5bb16bbbfa17256b0a9f56ca6ffd6daf6d7a33e"}
Sep 15 11:50:56.183261 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:56.182780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs" event={"ID":"f0a5e0c0-71ba-43aa-8272-91158e33853e","Type":"ContainerStarted","Data":"4f1aec8118f7b1a732bd1f2ba0f84fd8d76328102159204abd35571365a1d63b"}
Sep 15 11:50:56.203124 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:50:56.203074    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs" podStartSLOduration=1.9608124070000001 podStartE2EDuration="2.203058671s" totalImagesPullingTime="242.246264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:50:56.201848341 +0000 UTC m=+744.982419729" watchObservedRunningTime="2026-09-15 11:50:56.203058671 +0000 UTC m=+744.983630058"
Sep 15 11:51:04.982381 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:04.982343    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:51:04.982779 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:04.982424    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:51:04.994726 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:04.994702    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:51:05.240069 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:05.239997    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tvszs"
Sep 15 11:51:08.056071 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.056027    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"]
Sep 15 11:51:08.059134 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.059113    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.061398 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.061382    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 11:51:08.070044 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.070017    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"]
Sep 15 11:51:08.200574 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.200538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c187a7-7ffa-42a2-bf62-f9d437620759-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.200743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.200587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c187a7-7ffa-42a2-bf62-f9d437620759-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.200743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.200614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c187a7-7ffa-42a2-bf62-f9d437620759-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.200743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.200672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn7gk\" (UniqueName: \"kubernetes.io/projected/e5c187a7-7ffa-42a2-bf62-f9d437620759-kube-api-access-jn7gk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.200865 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.200758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c187a7-7ffa-42a2-bf62-f9d437620759-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.200865 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.200779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c187a7-7ffa-42a2-bf62-f9d437620759-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.302042 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.302006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c187a7-7ffa-42a2-bf62-f9d437620759-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.302042 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.302042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c187a7-7ffa-42a2-bf62-f9d437620759-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.302243 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.302070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c187a7-7ffa-42a2-bf62-f9d437620759-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.302243 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.302095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c187a7-7ffa-42a2-bf62-f9d437620759-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.302243 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.302112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c187a7-7ffa-42a2-bf62-f9d437620759-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.302243 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.302138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn7gk\" (UniqueName: \"kubernetes.io/projected/e5c187a7-7ffa-42a2-bf62-f9d437620759-kube-api-access-jn7gk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.302448 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.302429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c187a7-7ffa-42a2-bf62-f9d437620759-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.302512 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.302489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c187a7-7ffa-42a2-bf62-f9d437620759-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.302595 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.302572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c187a7-7ffa-42a2-bf62-f9d437620759-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.304478 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.304449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c187a7-7ffa-42a2-bf62-f9d437620759-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.304665 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.304649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c187a7-7ffa-42a2-bf62-f9d437620759-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.311311 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.311260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn7gk\" (UniqueName: \"kubernetes.io/projected/e5c187a7-7ffa-42a2-bf62-f9d437620759-kube-api-access-jn7gk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc\" (UID: \"e5c187a7-7ffa-42a2-bf62-f9d437620759\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.369261 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.369231    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:08.643615 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:08.643590    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"]
Sep 15 11:51:08.645249 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:51:08.645221    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5c187a7_7ffa_42a2_bf62_f9d437620759.slice/crio-3ba2337b54612f8d211781e505eef00fd81027a84ee924fa887dcf6a3dca0a6a WatchSource:0}: Error finding container 3ba2337b54612f8d211781e505eef00fd81027a84ee924fa887dcf6a3dca0a6a: Status 404 returned error can't find the container with id 3ba2337b54612f8d211781e505eef00fd81027a84ee924fa887dcf6a3dca0a6a
Sep 15 11:51:09.249684 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:09.249653    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc" event={"ID":"e5c187a7-7ffa-42a2-bf62-f9d437620759","Type":"ContainerStarted","Data":"f1b22e7fd51d60be3d5e3e346af03e4ebfa52c1afd811dc169170c476e118eea"}
Sep 15 11:51:09.250108 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:09.249694    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc" event={"ID":"e5c187a7-7ffa-42a2-bf62-f9d437620759","Type":"ContainerStarted","Data":"3ba2337b54612f8d211781e505eef00fd81027a84ee924fa887dcf6a3dca0a6a"}
Sep 15 11:51:09.270647 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:09.270597    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc" podStartSLOduration=1.065573304 podStartE2EDuration="1.270580552s" totalImagesPullingTime="205.007248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:51:09.268048636 +0000 UTC m=+758.048620023" watchObservedRunningTime="2026-09-15 11:51:09.270580552 +0000 UTC m=+758.051151939"
Sep 15 11:51:18.370468 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:18.370436    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:18.370898 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:18.370495    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:18.382859 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:18.382840    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:51:19.316987 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:51:19.316960    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc"
Sep 15 11:52:48.937058 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:48.937025    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"]
Sep 15 11:52:48.940358 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:48.940337    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:48.942705 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:48.942683    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-29d630-openshift-default-dockercfg-s8zzm\""
Sep 15 11:52:48.957234 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:48.957210    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"]
Sep 15 11:52:49.027793 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.027764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-credential-socket\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.027944 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.027806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ntdk\" (UniqueName: \"kubernetes.io/projected/26fda038-f54a-4b23-84ff-419f2d7d0365-kube-api-access-7ntdk\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.027944 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.027904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-workload-certs\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.027944 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.027937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-token\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.028052 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.027997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-workload-socket\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.028052 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.028040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-envoy\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.028122 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.028067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-podinfo\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.028159 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.028112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-data\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.028193 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.028167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fda038-f54a-4b23-84ff-419f2d7d0365-istiod-ca-cert\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.129003 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.128970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-workload-certs\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.129163 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.129011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-token\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.129163 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.129043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-workload-socket\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.129163 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.129068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-envoy\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.129163 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.129095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-podinfo\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.129481 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.129454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-workload-certs\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.129616 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.129449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-data\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.129616 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.129582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fda038-f54a-4b23-84ff-419f2d7d0365-istiod-ca-cert\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.129616 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.129459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-workload-socket\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.129788 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.129664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-credential-socket\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.129788 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.129729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ntdk\" (UniqueName: \"kubernetes.io/projected/26fda038-f54a-4b23-84ff-419f2d7d0365-kube-api-access-7ntdk\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.129788 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.129745    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-data\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.130093 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.130048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-credential-socket\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.130266 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.130242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fda038-f54a-4b23-84ff-419f2d7d0365-istiod-ca-cert\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.131661 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.131630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-envoy\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.132018 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.131998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-podinfo\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.142980 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.142957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ntdk\" (UniqueName: \"kubernetes.io/projected/26fda038-f54a-4b23-84ff-419f2d7d0365-kube-api-access-7ntdk\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.143715 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.143700    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-token\") pod \"e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") " pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.251003 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.250921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:49.556965 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.556938    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"]
Sep 15 11:52:49.559210 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:52:49.559184    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fda038_f54a_4b23_84ff_419f2d7d0365.slice/crio-3f485901cbd39ca95e656a1bc3fe2e63fbd80c56327372f4a7c92bfab93c754f WatchSource:0}: Error finding container 3f485901cbd39ca95e656a1bc3fe2e63fbd80c56327372f4a7c92bfab93c754f: Status 404 returned error can't find the container with id 3f485901cbd39ca95e656a1bc3fe2e63fbd80c56327372f4a7c92bfab93c754f
Sep 15 11:52:49.561499 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.561462    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:52:49.561606 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.561535    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:52:49.561606 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.561570    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:52:49.757118 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.757077    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d" event={"ID":"26fda038-f54a-4b23-84ff-419f2d7d0365","Type":"ContainerStarted","Data":"35669b666b12877e06d96a8dd6acb91d79aa5e2e03e2561ec8875af4d1cb1d6c"}
Sep 15 11:52:49.757251 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.757140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d" event={"ID":"26fda038-f54a-4b23-84ff-419f2d7d0365","Type":"ContainerStarted","Data":"3f485901cbd39ca95e656a1bc3fe2e63fbd80c56327372f4a7c92bfab93c754f"}
Sep 15 11:52:49.790823 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:49.790749    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d" podStartSLOduration=1.79073421 podStartE2EDuration="1.79073421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:49.790205701 +0000 UTC m=+858.570777089" watchObservedRunningTime="2026-09-15 11:52:49.79073421 +0000 UTC m=+858.571305598"
Sep 15 11:52:50.251695 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:50.251667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:50.256905 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:50.256865    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:50.762269 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:50.762240    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:50.763254 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:50.763231    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:52:57.654716 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.654682    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"]
Sep 15 11:52:57.658366 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.658348    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.661350 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.661333    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-b42f08-openshift-default-dockercfg-zrknp\""
Sep 15 11:52:57.702242 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.702209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-token\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.702242 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.702243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istiod-ca-cert\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.702438 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.702331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gnpz\" (UniqueName: \"kubernetes.io/projected/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-kube-api-access-8gnpz\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.702438 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.702355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-podinfo\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.702438 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.702429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-workload-certs\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.702589 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.702451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-envoy\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.702589 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.702542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-credential-socket\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.702589 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.702575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-workload-socket\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.702734 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.702596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-data\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.748003 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.747973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"]
Sep 15 11:52:57.803195 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.803165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gnpz\" (UniqueName: \"kubernetes.io/projected/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-kube-api-access-8gnpz\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.803195 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.803200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-podinfo\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.803438 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.803231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-workload-certs\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.803438 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.803249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-envoy\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.803438 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.803307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-credential-socket\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.803438 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.803333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-workload-socket\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.803438 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.803361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-data\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.803438 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.803387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-token\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.803438 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.803402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istiod-ca-cert\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.803800 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.803643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-workload-certs\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.803860 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.803805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-data\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.803860 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.803829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-workload-socket\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.803984 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.803961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-credential-socket\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.804173 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.804150    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istiod-ca-cert\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.805687 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.805666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-envoy\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.805914 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.805895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-podinfo\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.831563 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.831533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-token\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.836778 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.836750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gnpz\" (UniqueName: \"kubernetes.io/projected/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-kube-api-access-8gnpz\") pod \"e2e-autores-b42f08-openshift-default-8f44cc586-skc9l\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") " pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:57.968683 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:57.968649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:58.276568 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:52:58.276531    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1af3bc7_bfa4_49b3_8ef4_1a9d2ddcff83.slice/crio-2dc603eca23c06104a7d685882d61ad5aa807f88375866d352793373190c2b78 WatchSource:0}: Error finding container 2dc603eca23c06104a7d685882d61ad5aa807f88375866d352793373190c2b78: Status 404 returned error can't find the container with id 2dc603eca23c06104a7d685882d61ad5aa807f88375866d352793373190c2b78
Sep 15 11:52:58.277112 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.277088    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"]
Sep 15 11:52:58.278670 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.278639    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:52:58.278754 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.278701    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:52:58.278754 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.278730    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:52:58.744110 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.744078    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"]
Sep 15 11:52:58.747583 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.747557    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"]
Sep 15 11:52:58.747742 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.747680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"
Sep 15 11:52:58.750843 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.750824    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"
Sep 15 11:52:58.757866 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.757846    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-97d02d\""
Sep 15 11:52:58.757866 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.757857    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-97d02d-dockercfg-hrmt8\""
Sep 15 11:52:58.769605 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.769582    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"]
Sep 15 11:52:58.772253 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.772233    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"]
Sep 15 11:52:58.802912 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.802857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l" event={"ID":"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83","Type":"ContainerStarted","Data":"2f90aa3b743ac160230dd433f51e3e34ab7d8cec92fc142dd69f59da084e43a9"}
Sep 15 11:52:58.803014 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.802928    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l" event={"ID":"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83","Type":"ContainerStarted","Data":"2dc603eca23c06104a7d685882d61ad5aa807f88375866d352793373190c2b78"}
Sep 15 11:52:58.812438 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.812411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz84r\" (UniqueName: \"kubernetes.io/projected/7871a402-06d0-4678-bd1e-1cdda46dd153-kube-api-access-tz84r\") pod \"payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q\" (UID: \"7871a402-06d0-4678-bd1e-1cdda46dd153\") " pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"
Sep 15 11:52:58.812583 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.812560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8598c\" (UniqueName: \"kubernetes.io/projected/e4b4b25f-8bda-4307-a7a5-b6d03a846ad7-kube-api-access-8598c\") pod \"payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq\" (UID: \"e4b4b25f-8bda-4307-a7a5-b6d03a846ad7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"
Sep 15 11:52:58.812649 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.812611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7871a402-06d0-4678-bd1e-1cdda46dd153-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q\" (UID: \"7871a402-06d0-4678-bd1e-1cdda46dd153\") " pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"
Sep 15 11:52:58.812649 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.812640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4b25f-8bda-4307-a7a5-b6d03a846ad7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq\" (UID: \"e4b4b25f-8bda-4307-a7a5-b6d03a846ad7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"
Sep 15 11:52:58.880191 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.880145    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l" podStartSLOduration=1.8801315060000001 podStartE2EDuration="1.880131506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:58.876966699 +0000 UTC m=+867.657538086" watchObservedRunningTime="2026-09-15 11:52:58.880131506 +0000 UTC m=+867.660702894"
Sep 15 11:52:58.913511 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.913483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz84r\" (UniqueName: \"kubernetes.io/projected/7871a402-06d0-4678-bd1e-1cdda46dd153-kube-api-access-tz84r\") pod \"payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q\" (UID: \"7871a402-06d0-4678-bd1e-1cdda46dd153\") " pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"
Sep 15 11:52:58.913628 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.913520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8598c\" (UniqueName: \"kubernetes.io/projected/e4b4b25f-8bda-4307-a7a5-b6d03a846ad7-kube-api-access-8598c\") pod \"payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq\" (UID: \"e4b4b25f-8bda-4307-a7a5-b6d03a846ad7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"
Sep 15 11:52:58.913896 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.913850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7871a402-06d0-4678-bd1e-1cdda46dd153-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q\" (UID: \"7871a402-06d0-4678-bd1e-1cdda46dd153\") " pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"
Sep 15 11:52:58.913978 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.913900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4b25f-8bda-4307-a7a5-b6d03a846ad7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq\" (UID: \"e4b4b25f-8bda-4307-a7a5-b6d03a846ad7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"
Sep 15 11:52:58.914513 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.914492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7871a402-06d0-4678-bd1e-1cdda46dd153-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q\" (UID: \"7871a402-06d0-4678-bd1e-1cdda46dd153\") " pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"
Sep 15 11:52:58.914734 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.914716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4b25f-8bda-4307-a7a5-b6d03a846ad7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq\" (UID: \"e4b4b25f-8bda-4307-a7a5-b6d03a846ad7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"
Sep 15 11:52:58.927702 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.927684    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz84r\" (UniqueName: \"kubernetes.io/projected/7871a402-06d0-4678-bd1e-1cdda46dd153-kube-api-access-tz84r\") pod \"payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q\" (UID: \"7871a402-06d0-4678-bd1e-1cdda46dd153\") " pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"
Sep 15 11:52:58.946492 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.946466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8598c\" (UniqueName: \"kubernetes.io/projected/e4b4b25f-8bda-4307-a7a5-b6d03a846ad7-kube-api-access-8598c\") pod \"payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq\" (UID: \"e4b4b25f-8bda-4307-a7a5-b6d03a846ad7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"
Sep 15 11:52:58.969267 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:58.969243    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:52:59.059761 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:59.059680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"
Sep 15 11:52:59.064803 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:59.064783    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"
Sep 15 11:52:59.345033 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:59.345009    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"]
Sep 15 11:52:59.346490 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:52:59.346450    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7871a402_06d0_4678_bd1e_1cdda46dd153.slice/crio-ad28eccc97213f84cfe40f04dfa13a78c652bfd0488d996630c1fb6257f29d01 WatchSource:0}: Error finding container ad28eccc97213f84cfe40f04dfa13a78c652bfd0488d996630c1fb6257f29d01: Status 404 returned error can't find the container with id ad28eccc97213f84cfe40f04dfa13a78c652bfd0488d996630c1fb6257f29d01
Sep 15 11:52:59.352729 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:59.352703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"]
Sep 15 11:52:59.354532 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:52:59.354503    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b4b25f_8bda_4307_a7a5_b6d03a846ad7.slice/crio-c49466853e1d47cae0c4ba823da7739ae30bb26bdcf33e9d814aff67a07f46fc WatchSource:0}: Error finding container c49466853e1d47cae0c4ba823da7739ae30bb26bdcf33e9d814aff67a07f46fc: Status 404 returned error can't find the container with id c49466853e1d47cae0c4ba823da7739ae30bb26bdcf33e9d814aff67a07f46fc
Sep 15 11:52:59.810115 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:59.810080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq" event={"ID":"e4b4b25f-8bda-4307-a7a5-b6d03a846ad7","Type":"ContainerStarted","Data":"82f03b86614251775906a220a0c687cbf15ec7a2d5e50a899c551c27080447d0"}
Sep 15 11:52:59.810560 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:59.810148    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"
Sep 15 11:52:59.810560 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:59.810175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq" event={"ID":"e4b4b25f-8bda-4307-a7a5-b6d03a846ad7","Type":"ContainerStarted","Data":"c49466853e1d47cae0c4ba823da7739ae30bb26bdcf33e9d814aff67a07f46fc"}
Sep 15 11:52:59.811941 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:59.811914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q" event={"ID":"7871a402-06d0-4678-bd1e-1cdda46dd153","Type":"ContainerStarted","Data":"6f9e413166760100ccfc55302c2f5fc0c18b065be1876d0a22c08e0163fb8a9a"}
Sep 15 11:52:59.812057 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:59.811965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q" event={"ID":"7871a402-06d0-4678-bd1e-1cdda46dd153","Type":"ContainerStarted","Data":"ad28eccc97213f84cfe40f04dfa13a78c652bfd0488d996630c1fb6257f29d01"}
Sep 15 11:52:59.812057 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:59.811996    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"
Sep 15 11:52:59.840143 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:59.840098    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq" podStartSLOduration=1.840083092 podStartE2EDuration="1.840083092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:59.838624167 +0000 UTC m=+868.619195555" watchObservedRunningTime="2026-09-15 11:52:59.840083092 +0000 UTC m=+868.620654488"
Sep 15 11:52:59.871170 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:59.871129    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q" podStartSLOduration=1.8711149470000001 podStartE2EDuration="1.871114947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:59.869639112 +0000 UTC m=+868.650210492" watchObservedRunningTime="2026-09-15 11:52:59.871114947 +0000 UTC m=+868.651686334"
Sep 15 11:52:59.974128 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:52:59.974102    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:53:00.817845 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:00.817813    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:53:00.818794 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:00.818773    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:53:05.828640 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:05.828608    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"]
Sep 15 11:53:05.832269 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:05.832251    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"
Sep 15 11:53:05.834830 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:05.834814    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-b42f08-dockercfg-mhhfr\""
Sep 15 11:53:05.835033 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:05.835016    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-b42f08\""
Sep 15 11:53:05.850799 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:05.850769    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"]
Sep 15 11:53:05.989283 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:05.989248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxm8\" (UniqueName: \"kubernetes.io/projected/6e1297b5-44fa-45f9-813a-06382923ce8d-kube-api-access-tqxm8\") pod \"payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8\" (UID: \"6e1297b5-44fa-45f9-813a-06382923ce8d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"
Sep 15 11:53:05.989639 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:05.989402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1297b5-44fa-45f9-813a-06382923ce8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8\" (UID: \"6e1297b5-44fa-45f9-813a-06382923ce8d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"
Sep 15 11:53:06.090673 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:06.090573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1297b5-44fa-45f9-813a-06382923ce8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8\" (UID: \"6e1297b5-44fa-45f9-813a-06382923ce8d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"
Sep 15 11:53:06.090854 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:06.090696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxm8\" (UniqueName: \"kubernetes.io/projected/6e1297b5-44fa-45f9-813a-06382923ce8d-kube-api-access-tqxm8\") pod \"payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8\" (UID: \"6e1297b5-44fa-45f9-813a-06382923ce8d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"
Sep 15 11:53:06.091355 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:06.091334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1297b5-44fa-45f9-813a-06382923ce8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8\" (UID: \"6e1297b5-44fa-45f9-813a-06382923ce8d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"
Sep 15 11:53:06.102110 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:06.102087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxm8\" (UniqueName: \"kubernetes.io/projected/6e1297b5-44fa-45f9-813a-06382923ce8d-kube-api-access-tqxm8\") pod \"payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8\" (UID: \"6e1297b5-44fa-45f9-813a-06382923ce8d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"
Sep 15 11:53:06.146225 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:06.146197    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"
Sep 15 11:53:06.444199 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:06.444170    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"]
Sep 15 11:53:06.445222 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:53:06.445196    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1297b5_44fa_45f9_813a_06382923ce8d.slice/crio-3c03b5ca5e581fbf444a9dce089d003835a119883d4a1b5a9aac66eae3186544 WatchSource:0}: Error finding container 3c03b5ca5e581fbf444a9dce089d003835a119883d4a1b5a9aac66eae3186544: Status 404 returned error can't find the container with id 3c03b5ca5e581fbf444a9dce089d003835a119883d4a1b5a9aac66eae3186544
Sep 15 11:53:06.875623 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:06.875525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8" event={"ID":"6e1297b5-44fa-45f9-813a-06382923ce8d","Type":"ContainerStarted","Data":"d3d1fe34dc7c03ea7d9fdd3095a023ca248d7646087626050c4a3969e6a14e98"}
Sep 15 11:53:06.876237 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:06.876211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8" event={"ID":"6e1297b5-44fa-45f9-813a-06382923ce8d","Type":"ContainerStarted","Data":"3c03b5ca5e581fbf444a9dce089d003835a119883d4a1b5a9aac66eae3186544"}
Sep 15 11:53:06.877133 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:06.877113    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"
Sep 15 11:53:06.901495 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:06.901443    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8" podStartSLOduration=1.901428654 podStartE2EDuration="1.901428654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:06.898413304 +0000 UTC m=+875.678984692" watchObservedRunningTime="2026-09-15 11:53:06.901428654 +0000 UTC m=+875.682000078"
Sep 15 11:53:09.130040 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.130005    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"]
Sep 15 11:53:09.134365 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.134345    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"
Sep 15 11:53:09.137278 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.137251    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-814e2181-dockercfg-m8d48\""
Sep 15 11:53:09.138015 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.137995    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-814e2181\""
Sep 15 11:53:09.142259 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.142234    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"]
Sep 15 11:53:09.147021 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.147000    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"]
Sep 15 11:53:09.147149 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.147104    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"
Sep 15 11:53:09.156660 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.156635    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"]
Sep 15 11:53:09.220690 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.220653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b533a3-f873-433c-b530-ea550c25b332-plugins-config-volume\") pod \"payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt\" (UID: \"c5b533a3-f873-433c-b530-ea550c25b332\") " pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"
Sep 15 11:53:09.220920 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.220707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d823ed8-61d3-456f-953a-cfd6b1174da4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m\" (UID: \"1d823ed8-61d3-456f-953a-cfd6b1174da4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"
Sep 15 11:53:09.220920 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.220732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnt5v\" (UniqueName: \"kubernetes.io/projected/c5b533a3-f873-433c-b530-ea550c25b332-kube-api-access-rnt5v\") pod \"payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt\" (UID: \"c5b533a3-f873-433c-b530-ea550c25b332\") " pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"
Sep 15 11:53:09.220920 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.220804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9hrg\" (UniqueName: \"kubernetes.io/projected/1d823ed8-61d3-456f-953a-cfd6b1174da4-kube-api-access-v9hrg\") pod \"payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m\" (UID: \"1d823ed8-61d3-456f-953a-cfd6b1174da4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"
Sep 15 11:53:09.321416 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.321372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9hrg\" (UniqueName: \"kubernetes.io/projected/1d823ed8-61d3-456f-953a-cfd6b1174da4-kube-api-access-v9hrg\") pod \"payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m\" (UID: \"1d823ed8-61d3-456f-953a-cfd6b1174da4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"
Sep 15 11:53:09.321615 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.321443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b533a3-f873-433c-b530-ea550c25b332-plugins-config-volume\") pod \"payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt\" (UID: \"c5b533a3-f873-433c-b530-ea550c25b332\") " pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"
Sep 15 11:53:09.321615 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.321481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d823ed8-61d3-456f-953a-cfd6b1174da4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m\" (UID: \"1d823ed8-61d3-456f-953a-cfd6b1174da4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"
Sep 15 11:53:09.321615 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.321510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnt5v\" (UniqueName: \"kubernetes.io/projected/c5b533a3-f873-433c-b530-ea550c25b332-kube-api-access-rnt5v\") pod \"payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt\" (UID: \"c5b533a3-f873-433c-b530-ea550c25b332\") " pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"
Sep 15 11:53:09.322277 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.322252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d823ed8-61d3-456f-953a-cfd6b1174da4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m\" (UID: \"1d823ed8-61d3-456f-953a-cfd6b1174da4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"
Sep 15 11:53:09.322393 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.322304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b533a3-f873-433c-b530-ea550c25b332-plugins-config-volume\") pod \"payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt\" (UID: \"c5b533a3-f873-433c-b530-ea550c25b332\") " pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"
Sep 15 11:53:09.331037 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.331010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9hrg\" (UniqueName: \"kubernetes.io/projected/1d823ed8-61d3-456f-953a-cfd6b1174da4-kube-api-access-v9hrg\") pod \"payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m\" (UID: \"1d823ed8-61d3-456f-953a-cfd6b1174da4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"
Sep 15 11:53:09.331187 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.331170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnt5v\" (UniqueName: \"kubernetes.io/projected/c5b533a3-f873-433c-b530-ea550c25b332-kube-api-access-rnt5v\") pod \"payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt\" (UID: \"c5b533a3-f873-433c-b530-ea550c25b332\") " pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"
Sep 15 11:53:09.447885 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.447841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"
Sep 15 11:53:09.461069 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.461044    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"
Sep 15 11:53:09.737923 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:53:09.737894    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d823ed8_61d3_456f_953a_cfd6b1174da4.slice/crio-8cd342daf0b71e87eb99e1d859aad77387e2fac6cde8aedc7ae30331937db189 WatchSource:0}: Error finding container 8cd342daf0b71e87eb99e1d859aad77387e2fac6cde8aedc7ae30331937db189: Status 404 returned error can't find the container with id 8cd342daf0b71e87eb99e1d859aad77387e2fac6cde8aedc7ae30331937db189
Sep 15 11:53:09.738148 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.738118    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"]
Sep 15 11:53:09.765795 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.765769    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"]
Sep 15 11:53:09.767790 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:53:09.767766    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b533a3_f873_433c_b530_ea550c25b332.slice/crio-f2c8bfbba991fdaf0287a82c329bc646f660db368b7bafb172a6384d4f9f6264 WatchSource:0}: Error finding container f2c8bfbba991fdaf0287a82c329bc646f660db368b7bafb172a6384d4f9f6264: Status 404 returned error can't find the container with id f2c8bfbba991fdaf0287a82c329bc646f660db368b7bafb172a6384d4f9f6264
Sep 15 11:53:09.895999 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.895852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt" event={"ID":"c5b533a3-f873-433c-b530-ea550c25b332","Type":"ContainerStarted","Data":"ba1aaa87fd1a16bd135e8a3d2aeb9e2453bb0260bdd7a46998f0a6f4bf77b5e4"}
Sep 15 11:53:09.895999 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.895935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt" event={"ID":"c5b533a3-f873-433c-b530-ea550c25b332","Type":"ContainerStarted","Data":"f2c8bfbba991fdaf0287a82c329bc646f660db368b7bafb172a6384d4f9f6264"}
Sep 15 11:53:09.895999 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.895992    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"
Sep 15 11:53:09.897848 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.897822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m" event={"ID":"1d823ed8-61d3-456f-953a-cfd6b1174da4","Type":"ContainerStarted","Data":"f65de880ec166170817a65b6f1a709f5240153e0e3d8e45c552eedf85016e3e8"}
Sep 15 11:53:09.898008 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.897890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m" event={"ID":"1d823ed8-61d3-456f-953a-cfd6b1174da4","Type":"ContainerStarted","Data":"8cd342daf0b71e87eb99e1d859aad77387e2fac6cde8aedc7ae30331937db189"}
Sep 15 11:53:09.898008 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.897931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"
Sep 15 11:53:09.927096 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.927045    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt" podStartSLOduration=0.927025137 podStartE2EDuration="927.025137ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:09.925562053 +0000 UTC m=+878.706133446" watchObservedRunningTime="2026-09-15 11:53:09.927025137 +0000 UTC m=+878.707596527"
Sep 15 11:53:09.952358 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:09.952302    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m" podStartSLOduration=0.952281166 podStartE2EDuration="952.281166ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:09.95074731 +0000 UTC m=+878.731318699" watchObservedRunningTime="2026-09-15 11:53:09.952281166 +0000 UTC m=+878.732852555"
Sep 15 11:53:10.819057 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:10.819025    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"
Sep 15 11:53:10.819464 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:10.819385    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"
Sep 15 11:53:11.910697 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:11.910666    2580 generic.go:356] "Generic (PLEG): container finished" podID="26fda038-f54a-4b23-84ff-419f2d7d0365" containerID="35669b666b12877e06d96a8dd6acb91d79aa5e2e03e2561ec8875af4d1cb1d6c" exitCode=0
Sep 15 11:53:11.911161 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:11.910727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d" event={"ID":"26fda038-f54a-4b23-84ff-419f2d7d0365","Type":"ContainerDied","Data":"35669b666b12877e06d96a8dd6acb91d79aa5e2e03e2561ec8875af4d1cb1d6c"}
Sep 15 11:53:11.911208 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:11.911195    2580 scope.go:117] "RemoveContainer" containerID="35669b666b12877e06d96a8dd6acb91d79aa5e2e03e2561ec8875af4d1cb1d6c"
Sep 15 11:53:11.912423 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:11.912397    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:53:11.912484 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:11.912462    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:53:11.912530 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:11.912495    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:53:12.919646 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:12.919608    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d" event={"ID":"26fda038-f54a-4b23-84ff-419f2d7d0365","Type":"ContainerStarted","Data":"fb4ad45f91ed67060d8b001b5fdab8af349ebd28a83faac648fbefef59a06fd3"}
Sep 15 11:53:13.251770 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:13.251667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:53:13.251995 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:13.251795    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 11:53:13.251995 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:13.251856    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d" podUID="26fda038-f54a-4b23-84ff-419f2d7d0365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 11:53:14.251786 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:14.251324    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 11:53:14.251786 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:14.251388    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d" podUID="26fda038-f54a-4b23-84ff-419f2d7d0365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 11:53:15.251288 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:15.251252    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 11:53:15.251573 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:15.251544    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d" podUID="26fda038-f54a-4b23-84ff-419f2d7d0365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 11:53:16.251414 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:16.251372    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 11:53:16.252064 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:16.252026    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d" podUID="26fda038-f54a-4b23-84ff-419f2d7d0365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 11:53:17.251906 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:17.251396    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 11:53:17.251906 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:17.251477    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d" podUID="26fda038-f54a-4b23-84ff-419f2d7d0365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 11:53:18.258624 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:18.258591    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:53:18.259524 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:18.259488    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:53:18.261646 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:18.261621    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 11:53:18.889283 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:18.889251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"
Sep 15 11:53:20.906812 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:20.906781    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"
Sep 15 11:53:20.907395 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:20.906866    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"
Sep 15 11:53:21.748934 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.748849    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"]
Sep 15 11:53:21.751178 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.751162    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"
Sep 15 11:53:21.754471 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.754451    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e9bc87\""
Sep 15 11:53:21.757941 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.757781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee50cb3f-198a-41e3-876f-1ed5db6fdeae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2\" (UID: \"ee50cb3f-198a-41e3-876f-1ed5db6fdeae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"
Sep 15 11:53:21.757941 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.757826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smjpz\" (UniqueName: \"kubernetes.io/projected/ee50cb3f-198a-41e3-876f-1ed5db6fdeae-kube-api-access-smjpz\") pod \"payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2\" (UID: \"ee50cb3f-198a-41e3-876f-1ed5db6fdeae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"
Sep 15 11:53:21.769346 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.769323    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"]
Sep 15 11:53:21.858268 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.858234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smjpz\" (UniqueName: \"kubernetes.io/projected/ee50cb3f-198a-41e3-876f-1ed5db6fdeae-kube-api-access-smjpz\") pod \"payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2\" (UID: \"ee50cb3f-198a-41e3-876f-1ed5db6fdeae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"
Sep 15 11:53:21.858425 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.858334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee50cb3f-198a-41e3-876f-1ed5db6fdeae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2\" (UID: \"ee50cb3f-198a-41e3-876f-1ed5db6fdeae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"
Sep 15 11:53:21.859042 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.859022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee50cb3f-198a-41e3-876f-1ed5db6fdeae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2\" (UID: \"ee50cb3f-198a-41e3-876f-1ed5db6fdeae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"
Sep 15 11:53:21.879972 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.879944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smjpz\" (UniqueName: \"kubernetes.io/projected/ee50cb3f-198a-41e3-876f-1ed5db6fdeae-kube-api-access-smjpz\") pod \"payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2\" (UID: \"ee50cb3f-198a-41e3-876f-1ed5db6fdeae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"
Sep 15 11:53:21.975857 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.975825    2580 generic.go:356] "Generic (PLEG): container finished" podID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerID="2f90aa3b743ac160230dd433f51e3e34ab7d8cec92fc142dd69f59da084e43a9" exitCode=0
Sep 15 11:53:21.976231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.975909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l" event={"ID":"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83","Type":"ContainerDied","Data":"2f90aa3b743ac160230dd433f51e3e34ab7d8cec92fc142dd69f59da084e43a9"}
Sep 15 11:53:21.976348 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.976334    2580 scope.go:117] "RemoveContainer" containerID="2f90aa3b743ac160230dd433f51e3e34ab7d8cec92fc142dd69f59da084e43a9"
Sep 15 11:53:21.977422 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.977397    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:53:21.977480 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.977468    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:53:21.977518 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:21.977502    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:53:22.061617 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:22.061585    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"
Sep 15 11:53:22.350951 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:22.350923    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"]
Sep 15 11:53:22.352631 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:53:22.352590    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee50cb3f_198a_41e3_876f_1ed5db6fdeae.slice/crio-e456706533baba9e9509515d4d81cfceda7c83f83d479279bfe71d5f5f8daaaa WatchSource:0}: Error finding container e456706533baba9e9509515d4d81cfceda7c83f83d479279bfe71d5f5f8daaaa: Status 404 returned error can't find the container with id e456706533baba9e9509515d4d81cfceda7c83f83d479279bfe71d5f5f8daaaa
Sep 15 11:53:22.983221 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:22.983181    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2" event={"ID":"ee50cb3f-198a-41e3-876f-1ed5db6fdeae","Type":"ContainerStarted","Data":"bb62dac143fc8f743090fdd2a2e8cf16c6a430d08e28a91d53c1096e31fdd81e"}
Sep 15 11:53:22.983692 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:22.983240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2" event={"ID":"ee50cb3f-198a-41e3-876f-1ed5db6fdeae","Type":"ContainerStarted","Data":"e456706533baba9e9509515d4d81cfceda7c83f83d479279bfe71d5f5f8daaaa"}
Sep 15 11:53:22.983692 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:22.983379    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"
Sep 15 11:53:22.985466 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:22.985439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l" event={"ID":"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83","Type":"ContainerStarted","Data":"3dfe43f96e1db9d388689f87edf6b796f969519e7957c4d3949fcfded71212dd"}
Sep 15 11:53:23.004357 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:23.004310    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2" podStartSLOduration=2.004296048 podStartE2EDuration="2.004296048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:23.002726951 +0000 UTC m=+891.783298341" watchObservedRunningTime="2026-09-15 11:53:23.004296048 +0000 UTC m=+891.784867489"
Sep 15 11:53:23.969401 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:23.969299    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:53:23.969563 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:23.969436    2580 patch_prober.go:28] interesting pod/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 15 11:53:23.969563 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:23.969482    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l" podUID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 15 11:53:24.780710 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:24.780669    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m"]
Sep 15 11:53:24.783436 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:24.783411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m"
Sep 15 11:53:24.783889 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:24.783840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c536f60-6489-48a2-82ff-36cb45440814-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-814e2181-4vr5m\" (UID: \"8c536f60-6489-48a2-82ff-36cb45440814\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m"
Sep 15 11:53:24.785998 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:24.785973    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 11:53:24.786309 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:24.786289    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:53:24.797241 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:24.797215    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m"]
Sep 15 11:53:24.884422 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:24.884384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c536f60-6489-48a2-82ff-36cb45440814-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-814e2181-4vr5m\" (UID: \"8c536f60-6489-48a2-82ff-36cb45440814\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m"
Sep 15 11:53:24.885093 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:24.885067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c536f60-6489-48a2-82ff-36cb45440814-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-814e2181-4vr5m\" (UID: \"8c536f60-6489-48a2-82ff-36cb45440814\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m"
Sep 15 11:53:24.969646 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:24.969600    2580 patch_prober.go:28] interesting pod/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 15 11:53:24.969837 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:24.969670    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l" podUID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 15 11:53:25.095021 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:25.094933    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m"
Sep 15 11:53:25.380290 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:25.380241    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m"]
Sep 15 11:53:25.381171 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:53:25.381143    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c536f60_6489_48a2_82ff_36cb45440814.slice/crio-2fd9a192df17ae3a3ebeb539534a628a02ee00762dcdef1a2954636b416e042c WatchSource:0}: Error finding container 2fd9a192df17ae3a3ebeb539534a628a02ee00762dcdef1a2954636b416e042c: Status 404 returned error can't find the container with id 2fd9a192df17ae3a3ebeb539534a628a02ee00762dcdef1a2954636b416e042c
Sep 15 11:53:25.969324 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:25.969283    2580 patch_prober.go:28] interesting pod/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 15 11:53:25.969886 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:25.969355    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l" podUID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 15 11:53:26.002654 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:26.002616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m" event={"ID":"8c536f60-6489-48a2-82ff-36cb45440814","Type":"ContainerStarted","Data":"2fd9a192df17ae3a3ebeb539534a628a02ee00762dcdef1a2954636b416e042c"}
Sep 15 11:53:26.970112 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:26.970073    2580 patch_prober.go:28] interesting pod/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 15 11:53:26.970590 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:26.970144    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l" podUID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 15 11:53:27.972493 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:27.972453    2580 patch_prober.go:28] interesting pod/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 15 11:53:27.972937 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:27.972521    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l" podUID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 15 11:53:28.973006 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:28.972976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:53:28.973443 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:28.973258    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:53:28.974068 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:28.974049    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:53:29.026366 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:29.026309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m" event={"ID":"8c536f60-6489-48a2-82ff-36cb45440814","Type":"ContainerStarted","Data":"c6b2f9ce59641f4625f11dc99c505f1dc14ac903c36ac76bd1ced7ccb8f5b66f"}
Sep 15 11:53:29.044797 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:29.044685    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m" podStartSLOduration=2.042835988 podStartE2EDuration="5.044662403s" totalImagesPullingTime="3.001826415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:29.043571219 +0000 UTC m=+897.824142620" watchObservedRunningTime="2026-09-15 11:53:29.044662403 +0000 UTC m=+897.825233792"
Sep 15 11:53:30.033662 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:30.033629    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c536f60-6489-48a2-82ff-36cb45440814" containerID="c6b2f9ce59641f4625f11dc99c505f1dc14ac903c36ac76bd1ced7ccb8f5b66f" exitCode=0
Sep 15 11:53:30.034027 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:30.033705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m" event={"ID":"8c536f60-6489-48a2-82ff-36cb45440814","Type":"ContainerDied","Data":"c6b2f9ce59641f4625f11dc99c505f1dc14ac903c36ac76bd1ced7ccb8f5b66f"}
Sep 15 11:53:31.167069 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:31.167046    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m"
Sep 15 11:53:31.239143 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:31.239107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c536f60-6489-48a2-82ff-36cb45440814-maas-api-service-ca\") pod \"8c536f60-6489-48a2-82ff-36cb45440814\" (UID: \"8c536f60-6489-48a2-82ff-36cb45440814\") "
Sep 15 11:53:31.239460 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:31.239436    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c536f60-6489-48a2-82ff-36cb45440814-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c536f60-6489-48a2-82ff-36cb45440814" (UID: "8c536f60-6489-48a2-82ff-36cb45440814"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:53:31.340741 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:31.340665    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c536f60-6489-48a2-82ff-36cb45440814-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:53:32.047195 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:32.047115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m" event={"ID":"8c536f60-6489-48a2-82ff-36cb45440814","Type":"ContainerDied","Data":"2fd9a192df17ae3a3ebeb539534a628a02ee00762dcdef1a2954636b416e042c"}
Sep 15 11:53:32.047195 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:32.047164    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m"
Sep 15 11:53:32.047367 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:32.047168    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fd9a192df17ae3a3ebeb539534a628a02ee00762dcdef1a2954636b416e042c"
Sep 15 11:53:33.992999 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:33.992967    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"
Sep 15 11:53:35.499374 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.499343    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"]
Sep 15 11:53:35.499772 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.499548    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt" podUID="c5b533a3-f873-433c-b530-ea550c25b332" containerName="payload-processing" containerID="cri-o://ba1aaa87fd1a16bd135e8a3d2aeb9e2453bb0260bdd7a46998f0a6f4bf77b5e4" gracePeriod=30
Sep 15 11:53:35.506134 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.506107    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"]
Sep 15 11:53:35.506356 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.506324    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m" podUID="1d823ed8-61d3-456f-953a-cfd6b1174da4" containerName="payload-pre-processing" containerID="cri-o://f65de880ec166170817a65b6f1a709f5240153e0e3d8e45c552eedf85016e3e8" gracePeriod=30
Sep 15 11:53:35.755276 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.755220    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"
Sep 15 11:53:35.779607 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.779580    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b533a3-f873-433c-b530-ea550c25b332-plugins-config-volume\") pod \"c5b533a3-f873-433c-b530-ea550c25b332\" (UID: \"c5b533a3-f873-433c-b530-ea550c25b332\") "
Sep 15 11:53:35.779737 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.779632    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnt5v\" (UniqueName: \"kubernetes.io/projected/c5b533a3-f873-433c-b530-ea550c25b332-kube-api-access-rnt5v\") pod \"c5b533a3-f873-433c-b530-ea550c25b332\" (UID: \"c5b533a3-f873-433c-b530-ea550c25b332\") "
Sep 15 11:53:35.780058 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.780031    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b533a3-f873-433c-b530-ea550c25b332-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5b533a3-f873-433c-b530-ea550c25b332" (UID: "c5b533a3-f873-433c-b530-ea550c25b332"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:53:35.781932 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.781912    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b533a3-f873-433c-b530-ea550c25b332-kube-api-access-rnt5v" (OuterVolumeSpecName: "kube-api-access-rnt5v") pod "c5b533a3-f873-433c-b530-ea550c25b332" (UID: "c5b533a3-f873-433c-b530-ea550c25b332"). InnerVolumeSpecName "kube-api-access-rnt5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:53:35.789733 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.789709    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"
Sep 15 11:53:35.836558 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.836519    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m"]
Sep 15 11:53:35.838891 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.838852    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-814e2181-4vr5m"]
Sep 15 11:53:35.881045 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.881010    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9hrg\" (UniqueName: \"kubernetes.io/projected/1d823ed8-61d3-456f-953a-cfd6b1174da4-kube-api-access-v9hrg\") pod \"1d823ed8-61d3-456f-953a-cfd6b1174da4\" (UID: \"1d823ed8-61d3-456f-953a-cfd6b1174da4\") "
Sep 15 11:53:35.881157 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.881051    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d823ed8-61d3-456f-953a-cfd6b1174da4-plugins-config-volume\") pod \"1d823ed8-61d3-456f-953a-cfd6b1174da4\" (UID: \"1d823ed8-61d3-456f-953a-cfd6b1174da4\") "
Sep 15 11:53:35.881565 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.881533    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d823ed8-61d3-456f-953a-cfd6b1174da4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d823ed8-61d3-456f-953a-cfd6b1174da4" (UID: "1d823ed8-61d3-456f-953a-cfd6b1174da4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:53:35.881684 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.881554    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b533a3-f873-433c-b530-ea550c25b332-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:53:35.881684 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.881622    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnt5v\" (UniqueName: \"kubernetes.io/projected/c5b533a3-f873-433c-b530-ea550c25b332-kube-api-access-rnt5v\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:53:35.883258 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.883238    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d823ed8-61d3-456f-953a-cfd6b1174da4-kube-api-access-v9hrg" (OuterVolumeSpecName: "kube-api-access-v9hrg") pod "1d823ed8-61d3-456f-953a-cfd6b1174da4" (UID: "1d823ed8-61d3-456f-953a-cfd6b1174da4"). InnerVolumeSpecName "kube-api-access-v9hrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:53:35.982942 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.982909    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9hrg\" (UniqueName: \"kubernetes.io/projected/1d823ed8-61d3-456f-953a-cfd6b1174da4-kube-api-access-v9hrg\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:53:35.982942 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:35.982938    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d823ed8-61d3-456f-953a-cfd6b1174da4-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:53:36.071428 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.071316    2580 generic.go:356] "Generic (PLEG): container finished" podID="1d823ed8-61d3-456f-953a-cfd6b1174da4" containerID="f65de880ec166170817a65b6f1a709f5240153e0e3d8e45c552eedf85016e3e8" exitCode=0
Sep 15 11:53:36.071602 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.071580    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"
Sep 15 11:53:36.071740 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.071714    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m" event={"ID":"1d823ed8-61d3-456f-953a-cfd6b1174da4","Type":"ContainerDied","Data":"f65de880ec166170817a65b6f1a709f5240153e0e3d8e45c552eedf85016e3e8"}
Sep 15 11:53:36.071916 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.071809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m" event={"ID":"1d823ed8-61d3-456f-953a-cfd6b1174da4","Type":"ContainerDied","Data":"8cd342daf0b71e87eb99e1d859aad77387e2fac6cde8aedc7ae30331937db189"}
Sep 15 11:53:36.072003 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.071958    2580 scope.go:117] "RemoveContainer" containerID="f65de880ec166170817a65b6f1a709f5240153e0e3d8e45c552eedf85016e3e8"
Sep 15 11:53:36.073361 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.073331    2580 generic.go:356] "Generic (PLEG): container finished" podID="c5b533a3-f873-433c-b530-ea550c25b332" containerID="ba1aaa87fd1a16bd135e8a3d2aeb9e2453bb0260bdd7a46998f0a6f4bf77b5e4" exitCode=0
Sep 15 11:53:36.073440 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.073412    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"
Sep 15 11:53:36.073598 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.073567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt" event={"ID":"c5b533a3-f873-433c-b530-ea550c25b332","Type":"ContainerDied","Data":"ba1aaa87fd1a16bd135e8a3d2aeb9e2453bb0260bdd7a46998f0a6f4bf77b5e4"}
Sep 15 11:53:36.073722 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.073625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt" event={"ID":"c5b533a3-f873-433c-b530-ea550c25b332","Type":"ContainerDied","Data":"f2c8bfbba991fdaf0287a82c329bc646f660db368b7bafb172a6384d4f9f6264"}
Sep 15 11:53:36.085554 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.085372    2580 scope.go:117] "RemoveContainer" containerID="f65de880ec166170817a65b6f1a709f5240153e0e3d8e45c552eedf85016e3e8"
Sep 15 11:53:36.085741 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:53:36.085716    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65de880ec166170817a65b6f1a709f5240153e0e3d8e45c552eedf85016e3e8\": container with ID starting with f65de880ec166170817a65b6f1a709f5240153e0e3d8e45c552eedf85016e3e8 not found: ID does not exist" containerID="f65de880ec166170817a65b6f1a709f5240153e0e3d8e45c552eedf85016e3e8"
Sep 15 11:53:36.085828 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.085750    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65de880ec166170817a65b6f1a709f5240153e0e3d8e45c552eedf85016e3e8"} err="failed to get container status \"f65de880ec166170817a65b6f1a709f5240153e0e3d8e45c552eedf85016e3e8\": rpc error: code = NotFound desc = could not find container \"f65de880ec166170817a65b6f1a709f5240153e0e3d8e45c552eedf85016e3e8\": container with ID starting with f65de880ec166170817a65b6f1a709f5240153e0e3d8e45c552eedf85016e3e8 not found: ID does not exist"
Sep 15 11:53:36.085909 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.085862    2580 scope.go:117] "RemoveContainer" containerID="ba1aaa87fd1a16bd135e8a3d2aeb9e2453bb0260bdd7a46998f0a6f4bf77b5e4"
Sep 15 11:53:36.096328 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.096297    2580 scope.go:117] "RemoveContainer" containerID="ba1aaa87fd1a16bd135e8a3d2aeb9e2453bb0260bdd7a46998f0a6f4bf77b5e4"
Sep 15 11:53:36.096651 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:53:36.096626    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1aaa87fd1a16bd135e8a3d2aeb9e2453bb0260bdd7a46998f0a6f4bf77b5e4\": container with ID starting with ba1aaa87fd1a16bd135e8a3d2aeb9e2453bb0260bdd7a46998f0a6f4bf77b5e4 not found: ID does not exist" containerID="ba1aaa87fd1a16bd135e8a3d2aeb9e2453bb0260bdd7a46998f0a6f4bf77b5e4"
Sep 15 11:53:36.096807 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.096654    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1aaa87fd1a16bd135e8a3d2aeb9e2453bb0260bdd7a46998f0a6f4bf77b5e4"} err="failed to get container status \"ba1aaa87fd1a16bd135e8a3d2aeb9e2453bb0260bdd7a46998f0a6f4bf77b5e4\": rpc error: code = NotFound desc = could not find container \"ba1aaa87fd1a16bd135e8a3d2aeb9e2453bb0260bdd7a46998f0a6f4bf77b5e4\": container with ID starting with ba1aaa87fd1a16bd135e8a3d2aeb9e2453bb0260bdd7a46998f0a6f4bf77b5e4 not found: ID does not exist"
Sep 15 11:53:36.105425 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.105403    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"]
Sep 15 11:53:36.109706 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.109683    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-814e2181-fdfbd66bc-kwtnt"]
Sep 15 11:53:36.121490 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.121468    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"]
Sep 15 11:53:36.126500 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.126479    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-814e2181-66cf86dcfd-j789m"]
Sep 15 11:53:36.959788 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.959754    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"]
Sep 15 11:53:36.960346 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.960326    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5b533a3-f873-433c-b530-ea550c25b332" containerName="payload-processing"
Sep 15 11:53:36.960396 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.960351    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5b533a3-f873-433c-b530-ea550c25b332" containerName="payload-processing"
Sep 15 11:53:36.960396 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.960371    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c536f60-6489-48a2-82ff-36cb45440814" containerName="revoke-keys"
Sep 15 11:53:36.960396 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.960380    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c536f60-6489-48a2-82ff-36cb45440814" containerName="revoke-keys"
Sep 15 11:53:36.960396 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.960392    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d823ed8-61d3-456f-953a-cfd6b1174da4" containerName="payload-pre-processing"
Sep 15 11:53:36.960536 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.960401    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d823ed8-61d3-456f-953a-cfd6b1174da4" containerName="payload-pre-processing"
Sep 15 11:53:36.960536 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.960501    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d823ed8-61d3-456f-953a-cfd6b1174da4" containerName="payload-pre-processing"
Sep 15 11:53:36.960536 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.960515    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5b533a3-f873-433c-b530-ea550c25b332" containerName="payload-processing"
Sep 15 11:53:36.960536 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.960526    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c536f60-6489-48a2-82ff-36cb45440814" containerName="revoke-keys"
Sep 15 11:53:36.963694 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.963677    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:36.966131 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.966110    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-6347dad8-gw-openshift-default-dockercfg-v8mzp\""
Sep 15 11:53:36.977836 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.977813    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"]
Sep 15 11:53:36.992458 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.992436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-credential-socket\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:36.992538 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.992480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-workload-certs\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:36.992604 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.992535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-data\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:36.992604 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.992567    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-envoy\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:36.992604 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.992596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-token\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:36.992717 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.992614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-podinfo\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:36.992717 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.992632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz9v9\" (UniqueName: \"kubernetes.io/projected/9483a53f-fecc-46e6-bc07-f5f70848fab8-kube-api-access-hz9v9\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:36.992717 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.992675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-workload-socket\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:36.992717 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:36.992698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9483a53f-fecc-46e6-bc07-f5f70848fab8-istiod-ca-cert\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.093358 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.093326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-credential-socket\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.093548 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.093370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-workload-certs\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.093548 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.093400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-data\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.093548 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.093428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-envoy\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.093548 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.093442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-token\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.093548 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.093461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-podinfo\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.093548 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.093497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz9v9\" (UniqueName: \"kubernetes.io/projected/9483a53f-fecc-46e6-bc07-f5f70848fab8-kube-api-access-hz9v9\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.093955 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.093792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-credential-socket\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.093955 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.093831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-data\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.094064 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.093950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-workload-certs\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.094106 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.094084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-workload-socket\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.094152 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.094119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9483a53f-fecc-46e6-bc07-f5f70848fab8-istiod-ca-cert\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.094374 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.094352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-workload-socket\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.094711 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.094688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9483a53f-fecc-46e6-bc07-f5f70848fab8-istiod-ca-cert\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.096012 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.095994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-envoy\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.096100 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.096048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-podinfo\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.102280 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.102253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz9v9\" (UniqueName: \"kubernetes.io/projected/9483a53f-fecc-46e6-bc07-f5f70848fab8-kube-api-access-hz9v9\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.102381 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.102363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-token\") pod \"e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") " pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.273651 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.273564    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:37.559177 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:53:37.559140    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9483a53f_fecc_46e6_bc07_f5f70848fab8.slice/crio-1b7d92de4a7a875526a29bb83a8bfba0ad9433050c45feef2a5369bb227188bc WatchSource:0}: Error finding container 1b7d92de4a7a875526a29bb83a8bfba0ad9433050c45feef2a5369bb227188bc: Status 404 returned error can't find the container with id 1b7d92de4a7a875526a29bb83a8bfba0ad9433050c45feef2a5369bb227188bc
Sep 15 11:53:37.559337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.559316    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"]
Sep 15 11:53:37.561758 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.561730    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:53:37.561826 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.561793    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:53:37.561826 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.561820    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:53:37.731422 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.731388    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d823ed8-61d3-456f-953a-cfd6b1174da4" path="/var/lib/kubelet/pods/1d823ed8-61d3-456f-953a-cfd6b1174da4/volumes"
Sep 15 11:53:37.731757 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.731744    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c536f60-6489-48a2-82ff-36cb45440814" path="/var/lib/kubelet/pods/8c536f60-6489-48a2-82ff-36cb45440814/volumes"
Sep 15 11:53:37.732107 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:37.732092    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5b533a3-f873-433c-b530-ea550c25b332" path="/var/lib/kubelet/pods/c5b533a3-f873-433c-b530-ea550c25b332/volumes"
Sep 15 11:53:38.090647 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:38.090609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn" event={"ID":"9483a53f-fecc-46e6-bc07-f5f70848fab8","Type":"ContainerStarted","Data":"94b91e5f25cc909379c61dcfe313ba90935d916cccd1c151198f5202b60d3488"}
Sep 15 11:53:38.091186 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:38.090673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn" event={"ID":"9483a53f-fecc-46e6-bc07-f5f70848fab8","Type":"ContainerStarted","Data":"1b7d92de4a7a875526a29bb83a8bfba0ad9433050c45feef2a5369bb227188bc"}
Sep 15 11:53:38.112763 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:38.112715    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn" podStartSLOduration=2.112698223 podStartE2EDuration="2.112698223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:38.110982017 +0000 UTC m=+906.891553405" watchObservedRunningTime="2026-09-15 11:53:38.112698223 +0000 UTC m=+906.893269612"
Sep 15 11:53:38.273890 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:38.273837    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:38.279466 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:38.279442    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:39.096233 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:39.096194    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:39.097314 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:39.097296    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:53:44.001967 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.001934    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"]
Sep 15 11:53:44.005253 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.005231    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"
Sep 15 11:53:44.007847 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.007829    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-6347dad8-dockercfg-cjdgc\""
Sep 15 11:53:44.008171 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.008148    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-6347dad8\""
Sep 15 11:53:44.024810 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.024784    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"]
Sep 15 11:53:44.052405 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.052372    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"]
Sep 15 11:53:44.054777 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.054761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"
Sep 15 11:53:44.061790 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.061758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcce4d95-4d3b-4a20-8a02-34bf69d04097-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r\" (UID: \"dcce4d95-4d3b-4a20-8a02-34bf69d04097\") " pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"
Sep 15 11:53:44.061911 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.061844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs7s2\" (UniqueName: \"kubernetes.io/projected/dcce4d95-4d3b-4a20-8a02-34bf69d04097-kube-api-access-qs7s2\") pod \"payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r\" (UID: \"dcce4d95-4d3b-4a20-8a02-34bf69d04097\") " pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"
Sep 15 11:53:44.069942 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.069909    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"]
Sep 15 11:53:44.163116 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.163081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzzr2\" (UniqueName: \"kubernetes.io/projected/c24afa81-76a7-4517-8c05-19bb27dff6e8-kube-api-access-dzzr2\") pod \"payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8\" (UID: \"c24afa81-76a7-4517-8c05-19bb27dff6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"
Sep 15 11:53:44.163284 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.163156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcce4d95-4d3b-4a20-8a02-34bf69d04097-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r\" (UID: \"dcce4d95-4d3b-4a20-8a02-34bf69d04097\") " pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"
Sep 15 11:53:44.163284 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.163179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24afa81-76a7-4517-8c05-19bb27dff6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8\" (UID: \"c24afa81-76a7-4517-8c05-19bb27dff6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"
Sep 15 11:53:44.163284 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.163229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs7s2\" (UniqueName: \"kubernetes.io/projected/dcce4d95-4d3b-4a20-8a02-34bf69d04097-kube-api-access-qs7s2\") pod \"payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r\" (UID: \"dcce4d95-4d3b-4a20-8a02-34bf69d04097\") " pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"
Sep 15 11:53:44.163814 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.163796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcce4d95-4d3b-4a20-8a02-34bf69d04097-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r\" (UID: \"dcce4d95-4d3b-4a20-8a02-34bf69d04097\") " pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"
Sep 15 11:53:44.171936 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.171907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs7s2\" (UniqueName: \"kubernetes.io/projected/dcce4d95-4d3b-4a20-8a02-34bf69d04097-kube-api-access-qs7s2\") pod \"payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r\" (UID: \"dcce4d95-4d3b-4a20-8a02-34bf69d04097\") " pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"
Sep 15 11:53:44.264006 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.263910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzr2\" (UniqueName: \"kubernetes.io/projected/c24afa81-76a7-4517-8c05-19bb27dff6e8-kube-api-access-dzzr2\") pod \"payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8\" (UID: \"c24afa81-76a7-4517-8c05-19bb27dff6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"
Sep 15 11:53:44.264006 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.263995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24afa81-76a7-4517-8c05-19bb27dff6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8\" (UID: \"c24afa81-76a7-4517-8c05-19bb27dff6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"
Sep 15 11:53:44.264685 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.264664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24afa81-76a7-4517-8c05-19bb27dff6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8\" (UID: \"c24afa81-76a7-4517-8c05-19bb27dff6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"
Sep 15 11:53:44.272521 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.272499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzzr2\" (UniqueName: \"kubernetes.io/projected/c24afa81-76a7-4517-8c05-19bb27dff6e8-kube-api-access-dzzr2\") pod \"payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8\" (UID: \"c24afa81-76a7-4517-8c05-19bb27dff6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"
Sep 15 11:53:44.322322 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.322293    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"
Sep 15 11:53:44.365251 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.365216    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"
Sep 15 11:53:44.544825 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.544739    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"]
Sep 15 11:53:44.547866 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.547847    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.550206 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.550181    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-29d630\"/\"openshift-service-ca.crt\""
Sep 15 11:53:44.550323 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.550235    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-29d630\"/\"kube-root-ca.crt\""
Sep 15 11:53:44.550372 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.550342    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-29d630\"/\"default-dockercfg-s4dxr\""
Sep 15 11:53:44.550412 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.550374    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-29d630\"/\"simulator-w0-29d630-kserve-self-signed-certs\""
Sep 15 11:53:44.557170 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.557149    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"]
Sep 15 11:53:44.594591 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.594570    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"]
Sep 15 11:53:44.596419 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:53:44.596397    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcce4d95_4d3b_4a20_8a02_34bf69d04097.slice/crio-e6197a10f464354e5a43905b694590ad84a1959587300070372e1b2e4d5545c8 WatchSource:0}: Error finding container e6197a10f464354e5a43905b694590ad84a1959587300070372e1b2e4d5545c8: Status 404 returned error can't find the container with id e6197a10f464354e5a43905b694590ad84a1959587300070372e1b2e4d5545c8
Sep 15 11:53:44.648343 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.648315    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"]
Sep 15 11:53:44.649765 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:53:44.649740    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24afa81_76a7_4517_8c05_19bb27dff6e8.slice/crio-4f016ec561a72f2a33990405c76c2b421a8d4b72b4e6b828a48a4121f27b0097 WatchSource:0}: Error finding container 4f016ec561a72f2a33990405c76c2b421a8d4b72b4e6b828a48a4121f27b0097: Status 404 returned error can't find the container with id 4f016ec561a72f2a33990405c76c2b421a8d4b72b4e6b828a48a4121f27b0097
Sep 15 11:53:44.667942 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.667908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trrfq\" (UniqueName: \"kubernetes.io/projected/0767ba87-5a08-4c34-8df4-dbffd98dbb75-kube-api-access-trrfq\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.668062 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.668039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0767ba87-5a08-4c34-8df4-dbffd98dbb75-tls-certs\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.668103 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.668089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-model-cache\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.668175 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.668157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-dshm\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.668217 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.668191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-tmp-dir\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.668272 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.668218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-home\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.769093 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.769059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trrfq\" (UniqueName: \"kubernetes.io/projected/0767ba87-5a08-4c34-8df4-dbffd98dbb75-kube-api-access-trrfq\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.769289 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.769123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0767ba87-5a08-4c34-8df4-dbffd98dbb75-tls-certs\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.769289 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.769152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-model-cache\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.769289 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.769180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-dshm\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.769289 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.769198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-tmp-dir\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.769289 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.769225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-home\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.769636 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.769614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-home\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.769708 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.769642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-model-cache\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.769847 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.769832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-tmp-dir\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.771845 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.771819    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-dshm\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.772525 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.772505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0767ba87-5a08-4c34-8df4-dbffd98dbb75-tls-certs\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.779354 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.779330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trrfq\" (UniqueName: \"kubernetes.io/projected/0767ba87-5a08-4c34-8df4-dbffd98dbb75-kube-api-access-trrfq\") pod \"simulator-w0-29d630-kserve-7c5d9c975c-v98wb\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") " pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:44.858928 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:44.858825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:45.129926 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:45.129814    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8" event={"ID":"c24afa81-76a7-4517-8c05-19bb27dff6e8","Type":"ContainerStarted","Data":"a3314e7020fc473eecff8f7c65e3bf17eccd03cf2ead66eda4a28f2dd30ac64e"}
Sep 15 11:53:45.129926 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:45.129885    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8" event={"ID":"c24afa81-76a7-4517-8c05-19bb27dff6e8","Type":"ContainerStarted","Data":"4f016ec561a72f2a33990405c76c2b421a8d4b72b4e6b828a48a4121f27b0097"}
Sep 15 11:53:45.129926 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:45.129916    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"
Sep 15 11:53:45.131811 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:45.131727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r" event={"ID":"dcce4d95-4d3b-4a20-8a02-34bf69d04097","Type":"ContainerStarted","Data":"bb9dcd3ae5f0277a0e81d891f89572e76c4960cf944a98b78eaff1f987441fb2"}
Sep 15 11:53:45.132014 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:45.131838    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"
Sep 15 11:53:45.132014 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:45.131863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r" event={"ID":"dcce4d95-4d3b-4a20-8a02-34bf69d04097","Type":"ContainerStarted","Data":"e6197a10f464354e5a43905b694590ad84a1959587300070372e1b2e4d5545c8"}
Sep 15 11:53:45.135520 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:45.135498    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"]
Sep 15 11:53:45.137308 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:53:45.137283    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0767ba87_5a08_4c34_8df4_dbffd98dbb75.slice/crio-409a31e78a4b8c1a61f018db3a78cecc6431c3bdf63411806200f157b3403b25 WatchSource:0}: Error finding container 409a31e78a4b8c1a61f018db3a78cecc6431c3bdf63411806200f157b3403b25: Status 404 returned error can't find the container with id 409a31e78a4b8c1a61f018db3a78cecc6431c3bdf63411806200f157b3403b25
Sep 15 11:53:45.149059 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:45.149012    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8" podStartSLOduration=1.148994788 podStartE2EDuration="1.148994788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:45.146842102 +0000 UTC m=+913.927413492" watchObservedRunningTime="2026-09-15 11:53:45.148994788 +0000 UTC m=+913.929566178"
Sep 15 11:53:45.165506 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:45.165463    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r" podStartSLOduration=2.165449348 podStartE2EDuration="2.165449348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:45.164057314 +0000 UTC m=+913.944628704" watchObservedRunningTime="2026-09-15 11:53:45.165449348 +0000 UTC m=+913.946020736"
Sep 15 11:53:46.139663 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:46.139628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb" event={"ID":"0767ba87-5a08-4c34-8df4-dbffd98dbb75","Type":"ContainerStarted","Data":"339f9b1e29775abef418c7595b9ed61756f7e192e248fc28cb62c61b930beb1e"}
Sep 15 11:53:46.140110 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:46.139674    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb" event={"ID":"0767ba87-5a08-4c34-8df4-dbffd98dbb75","Type":"ContainerStarted","Data":"409a31e78a4b8c1a61f018db3a78cecc6431c3bdf63411806200f157b3403b25"}
Sep 15 11:53:46.160347 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:46.160296    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb" podStartSLOduration=2.160280612 podStartE2EDuration="2.160280612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:46.156989489 +0000 UTC m=+914.937560880" watchObservedRunningTime="2026-09-15 11:53:46.160280612 +0000 UTC m=+914.940852002"
Sep 15 11:53:54.859919 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:54.859866    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:54.860519 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:54.860499    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:54.880617 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:54.880583    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:55.203574 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:55.203542    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 11:53:56.141690 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:56.141661    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"
Sep 15 11:53:56.142203 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:53:56.141734    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"
Sep 15 11:54:02.235457 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:02.235375    2580 generic.go:356] "Generic (PLEG): container finished" podID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerID="94b91e5f25cc909379c61dcfe313ba90935d916cccd1c151198f5202b60d3488" exitCode=0
Sep 15 11:54:02.235845 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:02.235453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn" event={"ID":"9483a53f-fecc-46e6-bc07-f5f70848fab8","Type":"ContainerDied","Data":"94b91e5f25cc909379c61dcfe313ba90935d916cccd1c151198f5202b60d3488"}
Sep 15 11:54:02.235923 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:02.235867    2580 scope.go:117] "RemoveContainer" containerID="94b91e5f25cc909379c61dcfe313ba90935d916cccd1c151198f5202b60d3488"
Sep 15 11:54:02.237016 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:02.236989    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:54:02.237079 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:02.237056    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:54:02.237116 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:02.237088    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 11:54:03.242506 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:03.242471    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn" event={"ID":"9483a53f-fecc-46e6-bc07-f5f70848fab8","Type":"ContainerStarted","Data":"a7e796a0bc838dbf686ab81ea06afe4391da5994ca912e91ebbb065202c12448"}
Sep 15 11:54:03.277510 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:03.277196    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:54:03.277510 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:03.277367    2580 patch_prober.go:28] interesting pod/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 15 11:54:03.277510 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:03.277421    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn" podUID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 15 11:54:04.275286 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:04.275246    2580 patch_prober.go:28] interesting pod/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 15 11:54:04.275847 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:04.275816    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn" podUID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 15 11:54:05.274650 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:05.274618    2580 patch_prober.go:28] interesting pod/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 15 11:54:05.274891 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:05.274671    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn" podUID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 15 11:54:06.274246 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:06.274208    2580 patch_prober.go:28] interesting pod/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 15 11:54:06.274626 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:06.274281    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn" podUID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 15 11:54:07.274698 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:07.274661    2580 patch_prober.go:28] interesting pod/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 15 11:54:07.275113 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:07.274731    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn" podUID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 15 11:54:08.344319 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:08.344277    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:54:08.344783 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:08.344752    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:54:08.345915 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:08.345893    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:54:09.656455 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:09.656418    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt"]
Sep 15 11:54:09.659047 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:09.659029    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt"
Sep 15 11:54:09.661501 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:09.661479    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:54:09.661626 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:09.661479    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 11:54:09.668561 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:09.668541    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt"]
Sep 15 11:54:09.729518 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:09.729491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffdac088-b408-484d-ad28-88dfd66c58b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt\" (UID: \"ffdac088-b408-484d-ad28-88dfd66c58b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt"
Sep 15 11:54:09.830583 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:09.830552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffdac088-b408-484d-ad28-88dfd66c58b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt\" (UID: \"ffdac088-b408-484d-ad28-88dfd66c58b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt"
Sep 15 11:54:09.831197 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:09.831176    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffdac088-b408-484d-ad28-88dfd66c58b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt\" (UID: \"ffdac088-b408-484d-ad28-88dfd66c58b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt"
Sep 15 11:54:09.970401 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:09.970361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt"
Sep 15 11:54:10.244201 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:10.244178    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt"]
Sep 15 11:54:10.244940 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:54:10.244912    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdac088_b408_484d_ad28_88dfd66c58b7.slice/crio-cca36d09e5baaeffaaac4ef5acf405a7e4f0c14b1c886869b4d935a148b1fbf5 WatchSource:0}: Error finding container cca36d09e5baaeffaaac4ef5acf405a7e4f0c14b1c886869b4d935a148b1fbf5: Status 404 returned error can't find the container with id cca36d09e5baaeffaaac4ef5acf405a7e4f0c14b1c886869b4d935a148b1fbf5
Sep 15 11:54:10.302219 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:10.302191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt" event={"ID":"ffdac088-b408-484d-ad28-88dfd66c58b7","Type":"ContainerStarted","Data":"cca36d09e5baaeffaaac4ef5acf405a7e4f0c14b1c886869b4d935a148b1fbf5"}
Sep 15 11:54:11.310493 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:11.310399    2580 generic.go:356] "Generic (PLEG): container finished" podID="ffdac088-b408-484d-ad28-88dfd66c58b7" containerID="5b7df8ea8bb63407d30a71cdbebec17e9cb1c2c1a547f0ab02a6ba237a1a5e3a" exitCode=0
Sep 15 11:54:11.310493 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:11.310441    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt" event={"ID":"ffdac088-b408-484d-ad28-88dfd66c58b7","Type":"ContainerDied","Data":"5b7df8ea8bb63407d30a71cdbebec17e9cb1c2c1a547f0ab02a6ba237a1a5e3a"}
Sep 15 11:54:12.462138 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:12.462112    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt"
Sep 15 11:54:12.555222 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:12.555193    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffdac088-b408-484d-ad28-88dfd66c58b7-maas-api-service-ca\") pod \"ffdac088-b408-484d-ad28-88dfd66c58b7\" (UID: \"ffdac088-b408-484d-ad28-88dfd66c58b7\") "
Sep 15 11:54:12.555521 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:12.555498    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffdac088-b408-484d-ad28-88dfd66c58b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ffdac088-b408-484d-ad28-88dfd66c58b7" (UID: "ffdac088-b408-484d-ad28-88dfd66c58b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:54:12.656100 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:12.656026    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffdac088-b408-484d-ad28-88dfd66c58b7-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:13.323264 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:13.323234    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt"
Sep 15 11:54:13.323430 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:13.323267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt" event={"ID":"ffdac088-b408-484d-ad28-88dfd66c58b7","Type":"ContainerDied","Data":"cca36d09e5baaeffaaac4ef5acf405a7e4f0c14b1c886869b4d935a148b1fbf5"}
Sep 15 11:54:13.323430 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:13.323317    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cca36d09e5baaeffaaac4ef5acf405a7e4f0c14b1c886869b4d935a148b1fbf5"
Sep 15 11:54:20.967697 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:20.967654    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"]
Sep 15 11:54:20.968280 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:20.967860    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r" podUID="dcce4d95-4d3b-4a20-8a02-34bf69d04097" containerName="payload-processing" containerID="cri-o://bb9dcd3ae5f0277a0e81d891f89572e76c4960cf944a98b78eaff1f987441fb2" gracePeriod=30
Sep 15 11:54:20.973262 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:20.973036    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"]
Sep 15 11:54:20.973413 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:20.973241    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8" podUID="c24afa81-76a7-4517-8c05-19bb27dff6e8" containerName="payload-pre-processing" containerID="cri-o://a3314e7020fc473eecff8f7c65e3bf17eccd03cf2ead66eda4a28f2dd30ac64e" gracePeriod=30
Sep 15 11:54:21.150808 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.150748    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt"]
Sep 15 11:54:21.156340 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.156313    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6347dad8-fvmjt"]
Sep 15 11:54:21.248001 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.247975    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"
Sep 15 11:54:21.251078 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.251060    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"
Sep 15 11:54:21.292340 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.292282    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"]
Sep 15 11:54:21.292577 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.292537    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn" podUID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerName="istio-proxy" containerID="cri-o://a7e796a0bc838dbf686ab81ea06afe4391da5994ca912e91ebbb065202c12448" gracePeriod=30
Sep 15 11:54:21.341797 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.341763    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs7s2\" (UniqueName: \"kubernetes.io/projected/dcce4d95-4d3b-4a20-8a02-34bf69d04097-kube-api-access-qs7s2\") pod \"dcce4d95-4d3b-4a20-8a02-34bf69d04097\" (UID: \"dcce4d95-4d3b-4a20-8a02-34bf69d04097\") "
Sep 15 11:54:21.341982 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.341808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzzr2\" (UniqueName: \"kubernetes.io/projected/c24afa81-76a7-4517-8c05-19bb27dff6e8-kube-api-access-dzzr2\") pod \"c24afa81-76a7-4517-8c05-19bb27dff6e8\" (UID: \"c24afa81-76a7-4517-8c05-19bb27dff6e8\") "
Sep 15 11:54:21.341982 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.341883    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24afa81-76a7-4517-8c05-19bb27dff6e8-plugins-config-volume\") pod \"c24afa81-76a7-4517-8c05-19bb27dff6e8\" (UID: \"c24afa81-76a7-4517-8c05-19bb27dff6e8\") "
Sep 15 11:54:21.341982 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.341962    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcce4d95-4d3b-4a20-8a02-34bf69d04097-plugins-config-volume\") pod \"dcce4d95-4d3b-4a20-8a02-34bf69d04097\" (UID: \"dcce4d95-4d3b-4a20-8a02-34bf69d04097\") "
Sep 15 11:54:21.342322 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.342291    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24afa81-76a7-4517-8c05-19bb27dff6e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c24afa81-76a7-4517-8c05-19bb27dff6e8" (UID: "c24afa81-76a7-4517-8c05-19bb27dff6e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:54:21.342437 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.342337    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcce4d95-4d3b-4a20-8a02-34bf69d04097-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dcce4d95-4d3b-4a20-8a02-34bf69d04097" (UID: "dcce4d95-4d3b-4a20-8a02-34bf69d04097"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:54:21.343807 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.343787    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcce4d95-4d3b-4a20-8a02-34bf69d04097-kube-api-access-qs7s2" (OuterVolumeSpecName: "kube-api-access-qs7s2") pod "dcce4d95-4d3b-4a20-8a02-34bf69d04097" (UID: "dcce4d95-4d3b-4a20-8a02-34bf69d04097"). InnerVolumeSpecName "kube-api-access-qs7s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:54:21.344059 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.344040    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24afa81-76a7-4517-8c05-19bb27dff6e8-kube-api-access-dzzr2" (OuterVolumeSpecName: "kube-api-access-dzzr2") pod "c24afa81-76a7-4517-8c05-19bb27dff6e8" (UID: "c24afa81-76a7-4517-8c05-19bb27dff6e8"). InnerVolumeSpecName "kube-api-access-dzzr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:54:21.369439 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.369408    2580 generic.go:356] "Generic (PLEG): container finished" podID="c24afa81-76a7-4517-8c05-19bb27dff6e8" containerID="a3314e7020fc473eecff8f7c65e3bf17eccd03cf2ead66eda4a28f2dd30ac64e" exitCode=0
Sep 15 11:54:21.369559 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.369450    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"
Sep 15 11:54:21.369559 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.369497    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8" event={"ID":"c24afa81-76a7-4517-8c05-19bb27dff6e8","Type":"ContainerDied","Data":"a3314e7020fc473eecff8f7c65e3bf17eccd03cf2ead66eda4a28f2dd30ac64e"}
Sep 15 11:54:21.369559 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.369551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8" event={"ID":"c24afa81-76a7-4517-8c05-19bb27dff6e8","Type":"ContainerDied","Data":"4f016ec561a72f2a33990405c76c2b421a8d4b72b4e6b828a48a4121f27b0097"}
Sep 15 11:54:21.369727 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.369577    2580 scope.go:117] "RemoveContainer" containerID="a3314e7020fc473eecff8f7c65e3bf17eccd03cf2ead66eda4a28f2dd30ac64e"
Sep 15 11:54:21.370965 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.370941    2580 generic.go:356] "Generic (PLEG): container finished" podID="dcce4d95-4d3b-4a20-8a02-34bf69d04097" containerID="bb9dcd3ae5f0277a0e81d891f89572e76c4960cf944a98b78eaff1f987441fb2" exitCode=0
Sep 15 11:54:21.371077 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.370990    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"
Sep 15 11:54:21.371077 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.371001    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r" event={"ID":"dcce4d95-4d3b-4a20-8a02-34bf69d04097","Type":"ContainerDied","Data":"bb9dcd3ae5f0277a0e81d891f89572e76c4960cf944a98b78eaff1f987441fb2"}
Sep 15 11:54:21.371077 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.371053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r" event={"ID":"dcce4d95-4d3b-4a20-8a02-34bf69d04097","Type":"ContainerDied","Data":"e6197a10f464354e5a43905b694590ad84a1959587300070372e1b2e4d5545c8"}
Sep 15 11:54:21.380039 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.380020    2580 scope.go:117] "RemoveContainer" containerID="a3314e7020fc473eecff8f7c65e3bf17eccd03cf2ead66eda4a28f2dd30ac64e"
Sep 15 11:54:21.380352 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:54:21.380328    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3314e7020fc473eecff8f7c65e3bf17eccd03cf2ead66eda4a28f2dd30ac64e\": container with ID starting with a3314e7020fc473eecff8f7c65e3bf17eccd03cf2ead66eda4a28f2dd30ac64e not found: ID does not exist" containerID="a3314e7020fc473eecff8f7c65e3bf17eccd03cf2ead66eda4a28f2dd30ac64e"
Sep 15 11:54:21.380426 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.380359    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3314e7020fc473eecff8f7c65e3bf17eccd03cf2ead66eda4a28f2dd30ac64e"} err="failed to get container status \"a3314e7020fc473eecff8f7c65e3bf17eccd03cf2ead66eda4a28f2dd30ac64e\": rpc error: code = NotFound desc = could not find container \"a3314e7020fc473eecff8f7c65e3bf17eccd03cf2ead66eda4a28f2dd30ac64e\": container with ID starting with a3314e7020fc473eecff8f7c65e3bf17eccd03cf2ead66eda4a28f2dd30ac64e not found: ID does not exist"
Sep 15 11:54:21.380426 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.380380    2580 scope.go:117] "RemoveContainer" containerID="bb9dcd3ae5f0277a0e81d891f89572e76c4960cf944a98b78eaff1f987441fb2"
Sep 15 11:54:21.388416 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.388398    2580 scope.go:117] "RemoveContainer" containerID="bb9dcd3ae5f0277a0e81d891f89572e76c4960cf944a98b78eaff1f987441fb2"
Sep 15 11:54:21.388731 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:54:21.388709    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9dcd3ae5f0277a0e81d891f89572e76c4960cf944a98b78eaff1f987441fb2\": container with ID starting with bb9dcd3ae5f0277a0e81d891f89572e76c4960cf944a98b78eaff1f987441fb2 not found: ID does not exist" containerID="bb9dcd3ae5f0277a0e81d891f89572e76c4960cf944a98b78eaff1f987441fb2"
Sep 15 11:54:21.388823 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.388739    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9dcd3ae5f0277a0e81d891f89572e76c4960cf944a98b78eaff1f987441fb2"} err="failed to get container status \"bb9dcd3ae5f0277a0e81d891f89572e76c4960cf944a98b78eaff1f987441fb2\": rpc error: code = NotFound desc = could not find container \"bb9dcd3ae5f0277a0e81d891f89572e76c4960cf944a98b78eaff1f987441fb2\": container with ID starting with bb9dcd3ae5f0277a0e81d891f89572e76c4960cf944a98b78eaff1f987441fb2 not found: ID does not exist"
Sep 15 11:54:21.398800 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.398735    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"]
Sep 15 11:54:21.403797 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.403768    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6347dad8-6c997fc4bb-xqd9r"]
Sep 15 11:54:21.411938 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.411711    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"]
Sep 15 11:54:21.415430 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.415400    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6347dad8-578b5dffdb-tc4c8"]
Sep 15 11:54:21.443209 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.443186    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcce4d95-4d3b-4a20-8a02-34bf69d04097-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:21.443209 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.443208    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs7s2\" (UniqueName: \"kubernetes.io/projected/dcce4d95-4d3b-4a20-8a02-34bf69d04097-kube-api-access-qs7s2\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:21.443357 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.443218    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzzr2\" (UniqueName: \"kubernetes.io/projected/c24afa81-76a7-4517-8c05-19bb27dff6e8-kube-api-access-dzzr2\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:21.443357 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.443227    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24afa81-76a7-4517-8c05-19bb27dff6e8-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:21.731027 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.730994    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c24afa81-76a7-4517-8c05-19bb27dff6e8" path="/var/lib/kubelet/pods/c24afa81-76a7-4517-8c05-19bb27dff6e8/volumes"
Sep 15 11:54:21.731394 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.731382    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcce4d95-4d3b-4a20-8a02-34bf69d04097" path="/var/lib/kubelet/pods/dcce4d95-4d3b-4a20-8a02-34bf69d04097/volumes"
Sep 15 11:54:21.731669 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:21.731658    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffdac088-b408-484d-ad28-88dfd66c58b7" path="/var/lib/kubelet/pods/ffdac088-b408-484d-ad28-88dfd66c58b7/volumes"
Sep 15 11:54:23.346786 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:23.346743    2580 patch_prober.go:28] interesting pod/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 15 11:54:23.347247 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:23.346812    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn" podUID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 11:54:26.544751 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.544722    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:54:26.690223 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.690196    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz9v9\" (UniqueName: \"kubernetes.io/projected/9483a53f-fecc-46e6-bc07-f5f70848fab8-kube-api-access-hz9v9\") pod \"9483a53f-fecc-46e6-bc07-f5f70848fab8\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") "
Sep 15 11:54:26.690379 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.690253    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-envoy\") pod \"9483a53f-fecc-46e6-bc07-f5f70848fab8\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") "
Sep 15 11:54:26.690379 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.690287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-workload-socket\") pod \"9483a53f-fecc-46e6-bc07-f5f70848fab8\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") "
Sep 15 11:54:26.690379 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.690328    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-credential-socket\") pod \"9483a53f-fecc-46e6-bc07-f5f70848fab8\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") "
Sep 15 11:54:26.690379 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.690376    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-token\") pod \"9483a53f-fecc-46e6-bc07-f5f70848fab8\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") "
Sep 15 11:54:26.690580 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.690417    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-data\") pod \"9483a53f-fecc-46e6-bc07-f5f70848fab8\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") "
Sep 15 11:54:26.690580 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.690452    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-podinfo\") pod \"9483a53f-fecc-46e6-bc07-f5f70848fab8\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") "
Sep 15 11:54:26.690580 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.690482    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-workload-certs\") pod \"9483a53f-fecc-46e6-bc07-f5f70848fab8\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") "
Sep 15 11:54:26.690580 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.690507    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9483a53f-fecc-46e6-bc07-f5f70848fab8-istiod-ca-cert\") pod \"9483a53f-fecc-46e6-bc07-f5f70848fab8\" (UID: \"9483a53f-fecc-46e6-bc07-f5f70848fab8\") "
Sep 15 11:54:26.690787 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.690610    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9483a53f-fecc-46e6-bc07-f5f70848fab8" (UID: "9483a53f-fecc-46e6-bc07-f5f70848fab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:54:26.690787 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.690704    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9483a53f-fecc-46e6-bc07-f5f70848fab8" (UID: "9483a53f-fecc-46e6-bc07-f5f70848fab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:54:26.690955 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.690699    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-data" (OuterVolumeSpecName: "istio-data") pod "9483a53f-fecc-46e6-bc07-f5f70848fab8" (UID: "9483a53f-fecc-46e6-bc07-f5f70848fab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:54:26.690955 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.690948    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-workload-socket\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:26.691074 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.690968    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-credential-socket\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:26.691074 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.691011    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9483a53f-fecc-46e6-bc07-f5f70848fab8" (UID: "9483a53f-fecc-46e6-bc07-f5f70848fab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:54:26.691074 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.691042    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9483a53f-fecc-46e6-bc07-f5f70848fab8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9483a53f-fecc-46e6-bc07-f5f70848fab8" (UID: "9483a53f-fecc-46e6-bc07-f5f70848fab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:54:26.692907 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.692867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9483a53f-fecc-46e6-bc07-f5f70848fab8" (UID: "9483a53f-fecc-46e6-bc07-f5f70848fab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 11:54:26.692907 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.692893    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9483a53f-fecc-46e6-bc07-f5f70848fab8" (UID: "9483a53f-fecc-46e6-bc07-f5f70848fab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:54:26.693057 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.692906    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9483a53f-fecc-46e6-bc07-f5f70848fab8-kube-api-access-hz9v9" (OuterVolumeSpecName: "kube-api-access-hz9v9") pod "9483a53f-fecc-46e6-bc07-f5f70848fab8" (UID: "9483a53f-fecc-46e6-bc07-f5f70848fab8"). InnerVolumeSpecName "kube-api-access-hz9v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:54:26.693057 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.692924    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-token" (OuterVolumeSpecName: "istio-token") pod "9483a53f-fecc-46e6-bc07-f5f70848fab8" (UID: "9483a53f-fecc-46e6-bc07-f5f70848fab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:54:26.791756 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.791724    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz9v9\" (UniqueName: \"kubernetes.io/projected/9483a53f-fecc-46e6-bc07-f5f70848fab8-kube-api-access-hz9v9\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:26.791756 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.791754    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-envoy\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:26.791998 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.791774    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-token\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:26.791998 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.791785    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-data\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:26.791998 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.791797    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9483a53f-fecc-46e6-bc07-f5f70848fab8-istio-podinfo\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:26.791998 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.791808    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9483a53f-fecc-46e6-bc07-f5f70848fab8-workload-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:26.791998 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:26.791820    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9483a53f-fecc-46e6-bc07-f5f70848fab8-istiod-ca-cert\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:27.411097 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.411065    2580 generic.go:356] "Generic (PLEG): container finished" podID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerID="a7e796a0bc838dbf686ab81ea06afe4391da5994ca912e91ebbb065202c12448" exitCode=0
Sep 15 11:54:27.411252 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.411126    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"
Sep 15 11:54:27.411252 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.411152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn" event={"ID":"9483a53f-fecc-46e6-bc07-f5f70848fab8","Type":"ContainerDied","Data":"a7e796a0bc838dbf686ab81ea06afe4391da5994ca912e91ebbb065202c12448"}
Sep 15 11:54:27.411252 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.411238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn" event={"ID":"9483a53f-fecc-46e6-bc07-f5f70848fab8","Type":"ContainerDied","Data":"1b7d92de4a7a875526a29bb83a8bfba0ad9433050c45feef2a5369bb227188bc"}
Sep 15 11:54:27.411375 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.411268    2580 scope.go:117] "RemoveContainer" containerID="a7e796a0bc838dbf686ab81ea06afe4391da5994ca912e91ebbb065202c12448"
Sep 15 11:54:27.421326 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.421271    2580 scope.go:117] "RemoveContainer" containerID="94b91e5f25cc909379c61dcfe313ba90935d916cccd1c151198f5202b60d3488"
Sep 15 11:54:27.430070 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.430047    2580 scope.go:117] "RemoveContainer" containerID="a7e796a0bc838dbf686ab81ea06afe4391da5994ca912e91ebbb065202c12448"
Sep 15 11:54:27.430391 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:54:27.430370    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e796a0bc838dbf686ab81ea06afe4391da5994ca912e91ebbb065202c12448\": container with ID starting with a7e796a0bc838dbf686ab81ea06afe4391da5994ca912e91ebbb065202c12448 not found: ID does not exist" containerID="a7e796a0bc838dbf686ab81ea06afe4391da5994ca912e91ebbb065202c12448"
Sep 15 11:54:27.430458 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.430399    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e796a0bc838dbf686ab81ea06afe4391da5994ca912e91ebbb065202c12448"} err="failed to get container status \"a7e796a0bc838dbf686ab81ea06afe4391da5994ca912e91ebbb065202c12448\": rpc error: code = NotFound desc = could not find container \"a7e796a0bc838dbf686ab81ea06afe4391da5994ca912e91ebbb065202c12448\": container with ID starting with a7e796a0bc838dbf686ab81ea06afe4391da5994ca912e91ebbb065202c12448 not found: ID does not exist"
Sep 15 11:54:27.430458 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.430415    2580 scope.go:117] "RemoveContainer" containerID="94b91e5f25cc909379c61dcfe313ba90935d916cccd1c151198f5202b60d3488"
Sep 15 11:54:27.430700 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:54:27.430681    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b91e5f25cc909379c61dcfe313ba90935d916cccd1c151198f5202b60d3488\": container with ID starting with 94b91e5f25cc909379c61dcfe313ba90935d916cccd1c151198f5202b60d3488 not found: ID does not exist" containerID="94b91e5f25cc909379c61dcfe313ba90935d916cccd1c151198f5202b60d3488"
Sep 15 11:54:27.430763 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.430708    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b91e5f25cc909379c61dcfe313ba90935d916cccd1c151198f5202b60d3488"} err="failed to get container status \"94b91e5f25cc909379c61dcfe313ba90935d916cccd1c151198f5202b60d3488\": rpc error: code = NotFound desc = could not find container \"94b91e5f25cc909379c61dcfe313ba90935d916cccd1c151198f5202b60d3488\": container with ID starting with 94b91e5f25cc909379c61dcfe313ba90935d916cccd1c151198f5202b60d3488 not found: ID does not exist"
Sep 15 11:54:27.432678 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.432660    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"]
Sep 15 11:54:27.433045 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433032    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24afa81-76a7-4517-8c05-19bb27dff6e8" containerName="payload-pre-processing"
Sep 15 11:54:27.433089 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433047    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24afa81-76a7-4517-8c05-19bb27dff6e8" containerName="payload-pre-processing"
Sep 15 11:54:27.433089 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433062    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerName="istio-proxy"
Sep 15 11:54:27.433089 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433068    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerName="istio-proxy"
Sep 15 11:54:27.433089 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433075    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerName="istio-proxy"
Sep 15 11:54:27.433089 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433080    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerName="istio-proxy"
Sep 15 11:54:27.433231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433096    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcce4d95-4d3b-4a20-8a02-34bf69d04097" containerName="payload-processing"
Sep 15 11:54:27.433231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433102    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcce4d95-4d3b-4a20-8a02-34bf69d04097" containerName="payload-processing"
Sep 15 11:54:27.433231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433110    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffdac088-b408-484d-ad28-88dfd66c58b7" containerName="revoke-keys"
Sep 15 11:54:27.433231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433115    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffdac088-b408-484d-ad28-88dfd66c58b7" containerName="revoke-keys"
Sep 15 11:54:27.433231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433167    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerName="istio-proxy"
Sep 15 11:54:27.433231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433173    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcce4d95-4d3b-4a20-8a02-34bf69d04097" containerName="payload-processing"
Sep 15 11:54:27.433231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433182    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9483a53f-fecc-46e6-bc07-f5f70848fab8" containerName="istio-proxy"
Sep 15 11:54:27.433231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433188    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffdac088-b408-484d-ad28-88dfd66c58b7" containerName="revoke-keys"
Sep 15 11:54:27.433231 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.433195    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c24afa81-76a7-4517-8c05-19bb27dff6e8" containerName="payload-pre-processing"
Sep 15 11:54:27.436086 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.436071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"
Sep 15 11:54:27.438939 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.438920    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-917f4c86\""
Sep 15 11:54:27.441848 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.441827    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"]
Sep 15 11:54:27.447465 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.447425    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"]
Sep 15 11:54:27.450366 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.450347    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-6347dad8-gw-openshift-default-85895b5f7b-xx7hn"]
Sep 15 11:54:27.507359 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.507335    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"]
Sep 15 11:54:27.509754 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.509738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"
Sep 15 11:54:27.512244 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.512226    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-917f4c86-dockercfg-xrx5g\""
Sep 15 11:54:27.521173 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.521153    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"]
Sep 15 11:54:27.600457 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.600422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7575\" (UniqueName: \"kubernetes.io/projected/7cd92343-c5d5-4821-a9d2-d6010c47d0ba-kube-api-access-j7575\") pod \"payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj\" (UID: \"7cd92343-c5d5-4821-a9d2-d6010c47d0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"
Sep 15 11:54:27.600901 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.600472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b9nm\" (UniqueName: \"kubernetes.io/projected/8a9b46e2-9ae1-4dc0-a9c3-31e33162169d-kube-api-access-2b9nm\") pod \"payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v\" (UID: \"8a9b46e2-9ae1-4dc0-a9c3-31e33162169d\") " pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"
Sep 15 11:54:27.600901 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.600539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9b46e2-9ae1-4dc0-a9c3-31e33162169d-plugins-config-volume\") pod \"payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v\" (UID: \"8a9b46e2-9ae1-4dc0-a9c3-31e33162169d\") " pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"
Sep 15 11:54:27.600901 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.600609    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd92343-c5d5-4821-a9d2-d6010c47d0ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj\" (UID: \"7cd92343-c5d5-4821-a9d2-d6010c47d0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"
Sep 15 11:54:27.701802 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.701773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7575\" (UniqueName: \"kubernetes.io/projected/7cd92343-c5d5-4821-a9d2-d6010c47d0ba-kube-api-access-j7575\") pod \"payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj\" (UID: \"7cd92343-c5d5-4821-a9d2-d6010c47d0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"
Sep 15 11:54:27.701972 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.701810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b9nm\" (UniqueName: \"kubernetes.io/projected/8a9b46e2-9ae1-4dc0-a9c3-31e33162169d-kube-api-access-2b9nm\") pod \"payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v\" (UID: \"8a9b46e2-9ae1-4dc0-a9c3-31e33162169d\") " pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"
Sep 15 11:54:27.701972 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.701837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9b46e2-9ae1-4dc0-a9c3-31e33162169d-plugins-config-volume\") pod \"payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v\" (UID: \"8a9b46e2-9ae1-4dc0-a9c3-31e33162169d\") " pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"
Sep 15 11:54:27.701972 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.701866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd92343-c5d5-4821-a9d2-d6010c47d0ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj\" (UID: \"7cd92343-c5d5-4821-a9d2-d6010c47d0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"
Sep 15 11:54:27.702590 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.702570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9b46e2-9ae1-4dc0-a9c3-31e33162169d-plugins-config-volume\") pod \"payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v\" (UID: \"8a9b46e2-9ae1-4dc0-a9c3-31e33162169d\") " pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"
Sep 15 11:54:27.702645 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.702579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd92343-c5d5-4821-a9d2-d6010c47d0ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj\" (UID: \"7cd92343-c5d5-4821-a9d2-d6010c47d0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"
Sep 15 11:54:27.713533 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.713508    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b9nm\" (UniqueName: \"kubernetes.io/projected/8a9b46e2-9ae1-4dc0-a9c3-31e33162169d-kube-api-access-2b9nm\") pod \"payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v\" (UID: \"8a9b46e2-9ae1-4dc0-a9c3-31e33162169d\") " pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"
Sep 15 11:54:27.714002 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.713986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7575\" (UniqueName: \"kubernetes.io/projected/7cd92343-c5d5-4821-a9d2-d6010c47d0ba-kube-api-access-j7575\") pod \"payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj\" (UID: \"7cd92343-c5d5-4821-a9d2-d6010c47d0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"
Sep 15 11:54:27.731671 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.731647    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9483a53f-fecc-46e6-bc07-f5f70848fab8" path="/var/lib/kubelet/pods/9483a53f-fecc-46e6-bc07-f5f70848fab8/volumes"
Sep 15 11:54:27.747737 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.747713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"
Sep 15 11:54:27.820337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:27.820310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"
Sep 15 11:54:28.019142 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:28.019119    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"]
Sep 15 11:54:28.020309 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:54:28.020285    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9b46e2_9ae1_4dc0_a9c3_31e33162169d.slice/crio-06b889318afbf2c70174a0231799b4c79c06be30476fc76507b85b033f6da113 WatchSource:0}: Error finding container 06b889318afbf2c70174a0231799b4c79c06be30476fc76507b85b033f6da113: Status 404 returned error can't find the container with id 06b889318afbf2c70174a0231799b4c79c06be30476fc76507b85b033f6da113
Sep 15 11:54:28.095977 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:28.095952    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"]
Sep 15 11:54:28.097316 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:54:28.097266    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd92343_c5d5_4821_a9d2_d6010c47d0ba.slice/crio-367ff0d834634220a0c6fd36af5ef3fed2ca2a1f12ae2bdeda3f0a050914d7a4 WatchSource:0}: Error finding container 367ff0d834634220a0c6fd36af5ef3fed2ca2a1f12ae2bdeda3f0a050914d7a4: Status 404 returned error can't find the container with id 367ff0d834634220a0c6fd36af5ef3fed2ca2a1f12ae2bdeda3f0a050914d7a4
Sep 15 11:54:28.420354 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:28.420316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj" event={"ID":"7cd92343-c5d5-4821-a9d2-d6010c47d0ba","Type":"ContainerStarted","Data":"ac78c92061af7cd4f38a4e7973a06b0d8bd348b16a475f5a3f2ff116e0a9ace4"}
Sep 15 11:54:28.420548 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:28.420380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj" event={"ID":"7cd92343-c5d5-4821-a9d2-d6010c47d0ba","Type":"ContainerStarted","Data":"367ff0d834634220a0c6fd36af5ef3fed2ca2a1f12ae2bdeda3f0a050914d7a4"}
Sep 15 11:54:28.420548 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:28.420497    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"
Sep 15 11:54:28.421926 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:28.421901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v" event={"ID":"8a9b46e2-9ae1-4dc0-a9c3-31e33162169d","Type":"ContainerStarted","Data":"885404f27276de7b6951c6ed98b0f870256b3c335aa66cc91fdb98888458dd4e"}
Sep 15 11:54:28.422038 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:28.421946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v" event={"ID":"8a9b46e2-9ae1-4dc0-a9c3-31e33162169d","Type":"ContainerStarted","Data":"06b889318afbf2c70174a0231799b4c79c06be30476fc76507b85b033f6da113"}
Sep 15 11:54:28.422038 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:28.422029    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"
Sep 15 11:54:28.439189 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:28.439143    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj" podStartSLOduration=1.439128913 podStartE2EDuration="1.439128913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:54:28.437173815 +0000 UTC m=+957.217745217" watchObservedRunningTime="2026-09-15 11:54:28.439128913 +0000 UTC m=+957.219700300"
Sep 15 11:54:28.459343 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:28.459303    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v" podStartSLOduration=1.459291398 podStartE2EDuration="1.459291398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:54:28.457384946 +0000 UTC m=+957.237956339" watchObservedRunningTime="2026-09-15 11:54:28.459291398 +0000 UTC m=+957.239862819"
Sep 15 11:54:39.430689 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:39.430655    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"
Sep 15 11:54:39.431114 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:39.430742    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"
Sep 15 11:54:49.357083 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.357049    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"]
Sep 15 11:54:49.360894 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.360857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.364244 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.364207    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ac76e5\"/\"openshift-service-ca.crt\""
Sep 15 11:54:49.364244 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.364214    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac76e5\"/\"simulator-w4-ac76e5-kserve-self-signed-certs\""
Sep 15 11:54:49.364244 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.364207    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ac76e5\"/\"kube-root-ca.crt\""
Sep 15 11:54:49.364461 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.364244    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac76e5\"/\"default-dockercfg-mtvf9\""
Sep 15 11:54:49.369095 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.368964    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"]
Sep 15 11:54:49.394761 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.394737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-dshm\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.394905 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.394764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb869d8e-73ba-4545-956a-e42501dd8ec9-tls-certs\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.394905 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.394806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-home\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.394905 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.394840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-tmp-dir\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.395022 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.394950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-model-cache\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.395022 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.394971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw2z6\" (UniqueName: \"kubernetes.io/projected/fb869d8e-73ba-4545-956a-e42501dd8ec9-kube-api-access-cw2z6\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.495757 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.495719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-home\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.495757 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.495758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-tmp-dir\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.496015 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.495826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-model-cache\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.496015 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.495853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw2z6\" (UniqueName: \"kubernetes.io/projected/fb869d8e-73ba-4545-956a-e42501dd8ec9-kube-api-access-cw2z6\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.496015 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.495962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-dshm\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.496015 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.495988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb869d8e-73ba-4545-956a-e42501dd8ec9-tls-certs\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.496222 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.496130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-home\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.496222 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.496183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-model-cache\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.496326 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.496234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-tmp-dir\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.498188 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.498169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-dshm\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.498474 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.498455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb869d8e-73ba-4545-956a-e42501dd8ec9-tls-certs\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.508562 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.508542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw2z6\" (UniqueName: \"kubernetes.io/projected/fb869d8e-73ba-4545-956a-e42501dd8ec9-kube-api-access-cw2z6\") pod \"simulator-w4-ac76e5-kserve-84d48fb568-w9shj\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") " pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.671314 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.671283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:49.951836 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:49.951804    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"]
Sep 15 11:54:49.953243 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:54:49.953219    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb869d8e_73ba_4545_956a_e42501dd8ec9.slice/crio-a15a071d90d034476d8902c7693e20e000d747a4f5567c5f48126dd3a6c645b3 WatchSource:0}: Error finding container a15a071d90d034476d8902c7693e20e000d747a4f5567c5f48126dd3a6c645b3: Status 404 returned error can't find the container with id a15a071d90d034476d8902c7693e20e000d747a4f5567c5f48126dd3a6c645b3
Sep 15 11:54:50.554492 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:50.554451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj" event={"ID":"fb869d8e-73ba-4545-956a-e42501dd8ec9","Type":"ContainerStarted","Data":"34152fee890eafb7a862ba12b76f5f82ef3be38701c14e8ae432a49e0e4f53a3"}
Sep 15 11:54:50.554950 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:50.554502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj" event={"ID":"fb869d8e-73ba-4545-956a-e42501dd8ec9","Type":"ContainerStarted","Data":"a15a071d90d034476d8902c7693e20e000d747a4f5567c5f48126dd3a6c645b3"}
Sep 15 11:54:50.572464 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:50.572412    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj" podStartSLOduration=1.572398419 podStartE2EDuration="1.572398419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:54:50.571532976 +0000 UTC m=+979.352104366" watchObservedRunningTime="2026-09-15 11:54:50.572398419 +0000 UTC m=+979.352969806"
Sep 15 11:54:50.662610 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:50.662579    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6"]
Sep 15 11:54:50.665354 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:50.665336    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6"
Sep 15 11:54:50.667704 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:50.667672    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 11:54:50.667704 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:50.667701    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:54:50.673510 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:50.673482    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6"]
Sep 15 11:54:50.720521 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:50.720492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dba14c-9140-4626-a288-433146308daf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-917f4c86-brqg6\" (UID: \"c9dba14c-9140-4626-a288-433146308daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6"
Sep 15 11:54:50.821417 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:50.821343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dba14c-9140-4626-a288-433146308daf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-917f4c86-brqg6\" (UID: \"c9dba14c-9140-4626-a288-433146308daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6"
Sep 15 11:54:50.822060 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:50.822041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dba14c-9140-4626-a288-433146308daf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-917f4c86-brqg6\" (UID: \"c9dba14c-9140-4626-a288-433146308daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6"
Sep 15 11:54:50.978447 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:50.978414    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6"
Sep 15 11:54:51.252236 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:51.252205    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6"]
Sep 15 11:54:51.253859 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:54:51.253837    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9dba14c_9140_4626_a288_433146308daf.slice/crio-563173be7bedb63d96f61766167c9e69db0ee13ac663041ba691201f5097f77a WatchSource:0}: Error finding container 563173be7bedb63d96f61766167c9e69db0ee13ac663041ba691201f5097f77a: Status 404 returned error can't find the container with id 563173be7bedb63d96f61766167c9e69db0ee13ac663041ba691201f5097f77a
Sep 15 11:54:51.563198 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:51.563167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6" event={"ID":"c9dba14c-9140-4626-a288-433146308daf","Type":"ContainerStarted","Data":"563173be7bedb63d96f61766167c9e69db0ee13ac663041ba691201f5097f77a"}
Sep 15 11:54:52.569637 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:52.569602    2580 generic.go:356] "Generic (PLEG): container finished" podID="c9dba14c-9140-4626-a288-433146308daf" containerID="b43b80d6cc2a9bfa3b2625722d5fcdc784210632518842d3debc6727fe0adac8" exitCode=0
Sep 15 11:54:52.569988 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:52.569684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6" event={"ID":"c9dba14c-9140-4626-a288-433146308daf","Type":"ContainerDied","Data":"b43b80d6cc2a9bfa3b2625722d5fcdc784210632518842d3debc6727fe0adac8"}
Sep 15 11:54:53.705973 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:53.705951    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6"
Sep 15 11:54:53.750484 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:53.750461    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dba14c-9140-4626-a288-433146308daf-maas-api-service-ca\") pod \"c9dba14c-9140-4626-a288-433146308daf\" (UID: \"c9dba14c-9140-4626-a288-433146308daf\") "
Sep 15 11:54:53.750793 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:53.750769    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9dba14c-9140-4626-a288-433146308daf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c9dba14c-9140-4626-a288-433146308daf" (UID: "c9dba14c-9140-4626-a288-433146308daf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:54:53.852030 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:53.851950    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dba14c-9140-4626-a288-433146308daf-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:54.581464 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:54.581430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6" event={"ID":"c9dba14c-9140-4626-a288-433146308daf","Type":"ContainerDied","Data":"563173be7bedb63d96f61766167c9e69db0ee13ac663041ba691201f5097f77a"}
Sep 15 11:54:54.581645 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:54.581471    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6"
Sep 15 11:54:54.581645 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:54.581517    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="563173be7bedb63d96f61766167c9e69db0ee13ac663041ba691201f5097f77a"
Sep 15 11:54:56.688178 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:56.688147    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"]
Sep 15 11:54:56.688636 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:56.688365    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v" podUID="8a9b46e2-9ae1-4dc0-a9c3-31e33162169d" containerName="payload-processing" containerID="cri-o://885404f27276de7b6951c6ed98b0f870256b3c335aa66cc91fdb98888458dd4e" gracePeriod=30
Sep 15 11:54:56.693590 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:56.693560    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"]
Sep 15 11:54:56.693841 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:56.693795    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj" podUID="7cd92343-c5d5-4821-a9d2-d6010c47d0ba" containerName="payload-pre-processing" containerID="cri-o://ac78c92061af7cd4f38a4e7973a06b0d8bd348b16a475f5a3f2ff116e0a9ace4" gracePeriod=30
Sep 15 11:54:56.863753 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:56.863711    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6"]
Sep 15 11:54:56.866090 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:56.866067    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-917f4c86-brqg6"]
Sep 15 11:54:56.964203 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:56.964181    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"
Sep 15 11:54:56.967623 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:56.967605    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"
Sep 15 11:54:57.080230 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.080185    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b9nm\" (UniqueName: \"kubernetes.io/projected/8a9b46e2-9ae1-4dc0-a9c3-31e33162169d-kube-api-access-2b9nm\") pod \"8a9b46e2-9ae1-4dc0-a9c3-31e33162169d\" (UID: \"8a9b46e2-9ae1-4dc0-a9c3-31e33162169d\") "
Sep 15 11:54:57.080407 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.080266    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9b46e2-9ae1-4dc0-a9c3-31e33162169d-plugins-config-volume\") pod \"8a9b46e2-9ae1-4dc0-a9c3-31e33162169d\" (UID: \"8a9b46e2-9ae1-4dc0-a9c3-31e33162169d\") "
Sep 15 11:54:57.080407 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.080294    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd92343-c5d5-4821-a9d2-d6010c47d0ba-plugins-config-volume\") pod \"7cd92343-c5d5-4821-a9d2-d6010c47d0ba\" (UID: \"7cd92343-c5d5-4821-a9d2-d6010c47d0ba\") "
Sep 15 11:54:57.080407 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.080320    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7575\" (UniqueName: \"kubernetes.io/projected/7cd92343-c5d5-4821-a9d2-d6010c47d0ba-kube-api-access-j7575\") pod \"7cd92343-c5d5-4821-a9d2-d6010c47d0ba\" (UID: \"7cd92343-c5d5-4821-a9d2-d6010c47d0ba\") "
Sep 15 11:54:57.080656 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.080627    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd92343-c5d5-4821-a9d2-d6010c47d0ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7cd92343-c5d5-4821-a9d2-d6010c47d0ba" (UID: "7cd92343-c5d5-4821-a9d2-d6010c47d0ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:54:57.080656 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.080637    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9b46e2-9ae1-4dc0-a9c3-31e33162169d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a9b46e2-9ae1-4dc0-a9c3-31e33162169d" (UID: "8a9b46e2-9ae1-4dc0-a9c3-31e33162169d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:54:57.082427 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.082405    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd92343-c5d5-4821-a9d2-d6010c47d0ba-kube-api-access-j7575" (OuterVolumeSpecName: "kube-api-access-j7575") pod "7cd92343-c5d5-4821-a9d2-d6010c47d0ba" (UID: "7cd92343-c5d5-4821-a9d2-d6010c47d0ba"). InnerVolumeSpecName "kube-api-access-j7575". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:54:57.082519 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.082440    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9b46e2-9ae1-4dc0-a9c3-31e33162169d-kube-api-access-2b9nm" (OuterVolumeSpecName: "kube-api-access-2b9nm") pod "8a9b46e2-9ae1-4dc0-a9c3-31e33162169d" (UID: "8a9b46e2-9ae1-4dc0-a9c3-31e33162169d"). InnerVolumeSpecName "kube-api-access-2b9nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:54:57.181727 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.181697    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b9nm\" (UniqueName: \"kubernetes.io/projected/8a9b46e2-9ae1-4dc0-a9c3-31e33162169d-kube-api-access-2b9nm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:57.181727 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.181724    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9b46e2-9ae1-4dc0-a9c3-31e33162169d-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:57.181914 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.181739    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd92343-c5d5-4821-a9d2-d6010c47d0ba-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:57.181914 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.181751    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7575\" (UniqueName: \"kubernetes.io/projected/7cd92343-c5d5-4821-a9d2-d6010c47d0ba-kube-api-access-j7575\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:54:57.599720 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.599684    2580 generic.go:356] "Generic (PLEG): container finished" podID="7cd92343-c5d5-4821-a9d2-d6010c47d0ba" containerID="ac78c92061af7cd4f38a4e7973a06b0d8bd348b16a475f5a3f2ff116e0a9ace4" exitCode=0
Sep 15 11:54:57.599902 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.599729    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"
Sep 15 11:54:57.599902 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.599770    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj" event={"ID":"7cd92343-c5d5-4821-a9d2-d6010c47d0ba","Type":"ContainerDied","Data":"ac78c92061af7cd4f38a4e7973a06b0d8bd348b16a475f5a3f2ff116e0a9ace4"}
Sep 15 11:54:57.599902 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.599824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj" event={"ID":"7cd92343-c5d5-4821-a9d2-d6010c47d0ba","Type":"ContainerDied","Data":"367ff0d834634220a0c6fd36af5ef3fed2ca2a1f12ae2bdeda3f0a050914d7a4"}
Sep 15 11:54:57.599902 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.599844    2580 scope.go:117] "RemoveContainer" containerID="ac78c92061af7cd4f38a4e7973a06b0d8bd348b16a475f5a3f2ff116e0a9ace4"
Sep 15 11:54:57.601546 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.601516    2580 generic.go:356] "Generic (PLEG): container finished" podID="8a9b46e2-9ae1-4dc0-a9c3-31e33162169d" containerID="885404f27276de7b6951c6ed98b0f870256b3c335aa66cc91fdb98888458dd4e" exitCode=0
Sep 15 11:54:57.601692 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.601580    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"
Sep 15 11:54:57.601692 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.601603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v" event={"ID":"8a9b46e2-9ae1-4dc0-a9c3-31e33162169d","Type":"ContainerDied","Data":"885404f27276de7b6951c6ed98b0f870256b3c335aa66cc91fdb98888458dd4e"}
Sep 15 11:54:57.601692 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.601671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v" event={"ID":"8a9b46e2-9ae1-4dc0-a9c3-31e33162169d","Type":"ContainerDied","Data":"06b889318afbf2c70174a0231799b4c79c06be30476fc76507b85b033f6da113"}
Sep 15 11:54:57.610159 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.610142    2580 scope.go:117] "RemoveContainer" containerID="ac78c92061af7cd4f38a4e7973a06b0d8bd348b16a475f5a3f2ff116e0a9ace4"
Sep 15 11:54:57.610443 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:54:57.610419    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac78c92061af7cd4f38a4e7973a06b0d8bd348b16a475f5a3f2ff116e0a9ace4\": container with ID starting with ac78c92061af7cd4f38a4e7973a06b0d8bd348b16a475f5a3f2ff116e0a9ace4 not found: ID does not exist" containerID="ac78c92061af7cd4f38a4e7973a06b0d8bd348b16a475f5a3f2ff116e0a9ace4"
Sep 15 11:54:57.610553 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.610451    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac78c92061af7cd4f38a4e7973a06b0d8bd348b16a475f5a3f2ff116e0a9ace4"} err="failed to get container status \"ac78c92061af7cd4f38a4e7973a06b0d8bd348b16a475f5a3f2ff116e0a9ace4\": rpc error: code = NotFound desc = could not find container \"ac78c92061af7cd4f38a4e7973a06b0d8bd348b16a475f5a3f2ff116e0a9ace4\": container with ID starting with ac78c92061af7cd4f38a4e7973a06b0d8bd348b16a475f5a3f2ff116e0a9ace4 not found: ID does not exist"
Sep 15 11:54:57.610553 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.610473    2580 scope.go:117] "RemoveContainer" containerID="885404f27276de7b6951c6ed98b0f870256b3c335aa66cc91fdb98888458dd4e"
Sep 15 11:54:57.618773 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.618758    2580 scope.go:117] "RemoveContainer" containerID="885404f27276de7b6951c6ed98b0f870256b3c335aa66cc91fdb98888458dd4e"
Sep 15 11:54:57.619041 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:54:57.619026    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885404f27276de7b6951c6ed98b0f870256b3c335aa66cc91fdb98888458dd4e\": container with ID starting with 885404f27276de7b6951c6ed98b0f870256b3c335aa66cc91fdb98888458dd4e not found: ID does not exist" containerID="885404f27276de7b6951c6ed98b0f870256b3c335aa66cc91fdb98888458dd4e"
Sep 15 11:54:57.619095 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.619047    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885404f27276de7b6951c6ed98b0f870256b3c335aa66cc91fdb98888458dd4e"} err="failed to get container status \"885404f27276de7b6951c6ed98b0f870256b3c335aa66cc91fdb98888458dd4e\": rpc error: code = NotFound desc = could not find container \"885404f27276de7b6951c6ed98b0f870256b3c335aa66cc91fdb98888458dd4e\": container with ID starting with 885404f27276de7b6951c6ed98b0f870256b3c335aa66cc91fdb98888458dd4e not found: ID does not exist"
Sep 15 11:54:57.631510 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.631488    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"]
Sep 15 11:54:57.633882 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.633851    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-917f4c86-d5755c799-s9f4v"]
Sep 15 11:54:57.657531 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.657509    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"]
Sep 15 11:54:57.659668 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.659650    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-917f4c86-5576bb969d-gkkqj"]
Sep 15 11:54:57.731220 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.731194    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cd92343-c5d5-4821-a9d2-d6010c47d0ba" path="/var/lib/kubelet/pods/7cd92343-c5d5-4821-a9d2-d6010c47d0ba/volumes"
Sep 15 11:54:57.731550 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.731537    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a9b46e2-9ae1-4dc0-a9c3-31e33162169d" path="/var/lib/kubelet/pods/8a9b46e2-9ae1-4dc0-a9c3-31e33162169d/volumes"
Sep 15 11:54:57.731821 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:57.731809    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9dba14c-9140-4626-a288-433146308daf" path="/var/lib/kubelet/pods/c9dba14c-9140-4626-a288-433146308daf/volumes"
Sep 15 11:54:59.671645 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:59.671614    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:59.672122 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:59.671692    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:54:59.684665 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:54:59.684644    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:55:00.637541 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:00.637505    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 11:55:03.655892 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.655841    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"]
Sep 15 11:55:03.656273 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.656253    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd92343-c5d5-4821-a9d2-d6010c47d0ba" containerName="payload-pre-processing"
Sep 15 11:55:03.656323 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.656274    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd92343-c5d5-4821-a9d2-d6010c47d0ba" containerName="payload-pre-processing"
Sep 15 11:55:03.656323 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.656290    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9b46e2-9ae1-4dc0-a9c3-31e33162169d" containerName="payload-processing"
Sep 15 11:55:03.656323 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.656295    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9b46e2-9ae1-4dc0-a9c3-31e33162169d" containerName="payload-processing"
Sep 15 11:55:03.656323 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.656309    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9dba14c-9140-4626-a288-433146308daf" containerName="revoke-keys"
Sep 15 11:55:03.656323 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.656314    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9dba14c-9140-4626-a288-433146308daf" containerName="revoke-keys"
Sep 15 11:55:03.656476 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.656390    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9b46e2-9ae1-4dc0-a9c3-31e33162169d" containerName="payload-processing"
Sep 15 11:55:03.656476 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.656397    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd92343-c5d5-4821-a9d2-d6010c47d0ba" containerName="payload-pre-processing"
Sep 15 11:55:03.656476 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.656404    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9dba14c-9140-4626-a288-433146308daf" containerName="revoke-keys"
Sep 15 11:55:03.659566 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.659548    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"
Sep 15 11:55:03.661865 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.661844    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b407aeb7\""
Sep 15 11:55:03.661969 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.661937    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b407aeb7-dockercfg-2l26d\""
Sep 15 11:55:03.671175 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.671150    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"]
Sep 15 11:55:03.744551 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.744510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20da074-1db8-474d-b9fc-889a38ca5996-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp\" (UID: \"f20da074-1db8-474d-b9fc-889a38ca5996\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"
Sep 15 11:55:03.744821 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.744799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9hg9\" (UniqueName: \"kubernetes.io/projected/f20da074-1db8-474d-b9fc-889a38ca5996-kube-api-access-x9hg9\") pod \"payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp\" (UID: \"f20da074-1db8-474d-b9fc-889a38ca5996\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"
Sep 15 11:55:03.846168 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.846136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9hg9\" (UniqueName: \"kubernetes.io/projected/f20da074-1db8-474d-b9fc-889a38ca5996-kube-api-access-x9hg9\") pod \"payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp\" (UID: \"f20da074-1db8-474d-b9fc-889a38ca5996\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"
Sep 15 11:55:03.846363 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.846213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20da074-1db8-474d-b9fc-889a38ca5996-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp\" (UID: \"f20da074-1db8-474d-b9fc-889a38ca5996\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"
Sep 15 11:55:03.846957 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.846937    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20da074-1db8-474d-b9fc-889a38ca5996-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp\" (UID: \"f20da074-1db8-474d-b9fc-889a38ca5996\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"
Sep 15 11:55:03.855095 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.855073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9hg9\" (UniqueName: \"kubernetes.io/projected/f20da074-1db8-474d-b9fc-889a38ca5996-kube-api-access-x9hg9\") pod \"payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp\" (UID: \"f20da074-1db8-474d-b9fc-889a38ca5996\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"
Sep 15 11:55:03.971817 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:03.971776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"
Sep 15 11:55:04.248481 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:04.248452    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"]
Sep 15 11:55:04.250141 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:55:04.250112    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20da074_1db8_474d_b9fc_889a38ca5996.slice/crio-2fe9cb56b456d2e7a4a4e87c765e653ff8f5aaab4b05bc373d6b1e3c6d297fe4 WatchSource:0}: Error finding container 2fe9cb56b456d2e7a4a4e87c765e653ff8f5aaab4b05bc373d6b1e3c6d297fe4: Status 404 returned error can't find the container with id 2fe9cb56b456d2e7a4a4e87c765e653ff8f5aaab4b05bc373d6b1e3c6d297fe4
Sep 15 11:55:04.649683 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:04.649585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp" event={"ID":"f20da074-1db8-474d-b9fc-889a38ca5996","Type":"ContainerStarted","Data":"4b1a208ba70b71ed87605f27131098b91f2ae62b31a46a292fda32c34d99b9b2"}
Sep 15 11:55:04.649683 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:04.649654    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp" event={"ID":"f20da074-1db8-474d-b9fc-889a38ca5996","Type":"ContainerStarted","Data":"2fe9cb56b456d2e7a4a4e87c765e653ff8f5aaab4b05bc373d6b1e3c6d297fe4"}
Sep 15 11:55:04.649866 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:04.649686    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"
Sep 15 11:55:04.666857 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:04.666806    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp" podStartSLOduration=1.666790537 podStartE2EDuration="1.666790537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:55:04.665399971 +0000 UTC m=+993.445971370" watchObservedRunningTime="2026-09-15 11:55:04.666790537 +0000 UTC m=+993.447361965"
Sep 15 11:55:15.656836 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:15.656805    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"
Sep 15 11:55:16.260748 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.260713    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"]
Sep 15 11:55:16.264563 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.264534    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.266810 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.266788    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b42f08\"/\"autores-model-b42f08-kserve-self-signed-certs\""
Sep 15 11:55:16.267853 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.267828    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b42f08\"/\"kube-root-ca.crt\""
Sep 15 11:55:16.267983 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.267849    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b42f08\"/\"openshift-service-ca.crt\""
Sep 15 11:55:16.267983 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.267838    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b42f08\"/\"default-dockercfg-ztmbj\""
Sep 15 11:55:16.274251 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.274228    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"]
Sep 15 11:55:16.369191 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.369155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-home\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.369191 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.369193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hls6w\" (UniqueName: \"kubernetes.io/projected/83719daf-cd57-4189-b5a1-d28c4e999b8b-kube-api-access-hls6w\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.369392 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.369212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-model-cache\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.369392 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.369345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-tmp-dir\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.369505 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.369405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83719daf-cd57-4189-b5a1-d28c4e999b8b-tls-certs\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.369505 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.369465    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-dshm\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.470252 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.470222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-home\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.470252 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.470255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hls6w\" (UniqueName: \"kubernetes.io/projected/83719daf-cd57-4189-b5a1-d28c4e999b8b-kube-api-access-hls6w\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.470474 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.470273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-model-cache\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.470474 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.470322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-tmp-dir\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.470474 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.470361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83719daf-cd57-4189-b5a1-d28c4e999b8b-tls-certs\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.470474 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.470389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-dshm\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.470750 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.470727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-model-cache\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.470750 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.470744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-tmp-dir\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.470867 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.470791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-home\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.472718 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.472687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-dshm\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.473103 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.473077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83719daf-cd57-4189-b5a1-d28c4e999b8b-tls-certs\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.481183 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.481146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hls6w\" (UniqueName: \"kubernetes.io/projected/83719daf-cd57-4189-b5a1-d28c4e999b8b-kube-api-access-hls6w\") pod \"autores-model-b42f08-kserve-7fc9bd665f-gqtsp\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") " pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.576260 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.576171    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:16.849132 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:16.849106    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"]
Sep 15 11:55:16.850259 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:55:16.850229    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83719daf_cd57_4189_b5a1_d28c4e999b8b.slice/crio-cb731508b939c9836ad496dfaf740317f8f547782f0012b79efba96810ce784e WatchSource:0}: Error finding container cb731508b939c9836ad496dfaf740317f8f547782f0012b79efba96810ce784e: Status 404 returned error can't find the container with id cb731508b939c9836ad496dfaf740317f8f547782f0012b79efba96810ce784e
Sep 15 11:55:17.722585 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:17.722536    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp" event={"ID":"83719daf-cd57-4189-b5a1-d28c4e999b8b","Type":"ContainerStarted","Data":"632c566dc62862b44bef126ea349bc4e325d3a214d09240b38b141d2610e3b99"}
Sep 15 11:55:17.722743 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:17.722591    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp" event={"ID":"83719daf-cd57-4189-b5a1-d28c4e999b8b","Type":"ContainerStarted","Data":"cb731508b939c9836ad496dfaf740317f8f547782f0012b79efba96810ce784e"}
Sep 15 11:55:17.740765 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:17.740718    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp" podStartSLOduration=1.740701941 podStartE2EDuration="1.740701941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:55:17.739639449 +0000 UTC m=+1006.520210837" watchObservedRunningTime="2026-09-15 11:55:17.740701941 +0000 UTC m=+1006.521273329"
Sep 15 11:55:25.949907 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:25.949833    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd"]
Sep 15 11:55:25.953377 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:25.953358    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd"
Sep 15 11:55:25.955661 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:25.955642    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:55:25.955754 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:25.955733    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 11:55:25.961360 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:25.961333    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd"]
Sep 15 11:55:26.061314 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.061283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6b73c1-c76b-4a90-88a9-6c07780ad0fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd\" (UID: \"fb6b73c1-c76b-4a90-88a9-6c07780ad0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd"
Sep 15 11:55:26.161954 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.161918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6b73c1-c76b-4a90-88a9-6c07780ad0fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd\" (UID: \"fb6b73c1-c76b-4a90-88a9-6c07780ad0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd"
Sep 15 11:55:26.162611 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.162589    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6b73c1-c76b-4a90-88a9-6c07780ad0fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd\" (UID: \"fb6b73c1-c76b-4a90-88a9-6c07780ad0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd"
Sep 15 11:55:26.266588 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.266509    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd"
Sep 15 11:55:26.538054 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.538021    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd"]
Sep 15 11:55:26.539239 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:55:26.539213    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6b73c1_c76b_4a90_88a9_6c07780ad0fb.slice/crio-2b6e9608b29dd6265c6fec1b204832762fb785ac3e9746f0a51d144e822e5973 WatchSource:0}: Error finding container 2b6e9608b29dd6265c6fec1b204832762fb785ac3e9746f0a51d144e822e5973: Status 404 returned error can't find the container with id 2b6e9608b29dd6265c6fec1b204832762fb785ac3e9746f0a51d144e822e5973
Sep 15 11:55:26.576911 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.576859    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:26.577021 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.576926    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:26.589689 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.589668    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:26.659625 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.659588    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"]
Sep 15 11:55:26.664201 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.664180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.666533 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.666513    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-97d02d\"/\"test-model-97d02d-kserve-self-signed-certs\""
Sep 15 11:55:26.666533 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.666530    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-97d02d\"/\"default-dockercfg-bf6k2\""
Sep 15 11:55:26.666689 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.666550    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-97d02d\"/\"kube-root-ca.crt\""
Sep 15 11:55:26.666689 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.666592    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-97d02d\"/\"openshift-service-ca.crt\""
Sep 15 11:55:26.672361 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.672338    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"]
Sep 15 11:55:26.769179 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.769152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-home\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.769432 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.769407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-tmp-dir\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.769532 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.769454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-dshm\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.769577 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.769552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-tls-certs\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.769613 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.769584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxpq\" (UniqueName: \"kubernetes.io/projected/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-kube-api-access-tmxpq\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.769675 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.769655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-model-cache\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.774249 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.774222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd" event={"ID":"fb6b73c1-c76b-4a90-88a9-6c07780ad0fb","Type":"ContainerStarted","Data":"bdf7af673862def60dd8df3170deb4dbce7fbe93e06ade9f8ae73396b5d5655f"}
Sep 15 11:55:26.774337 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.774271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd" event={"ID":"fb6b73c1-c76b-4a90-88a9-6c07780ad0fb","Type":"ContainerStarted","Data":"2b6e9608b29dd6265c6fec1b204832762fb785ac3e9746f0a51d144e822e5973"}
Sep 15 11:55:26.785637 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.785613    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:26.800795 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.800720    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd" podStartSLOduration=1.80070676 podStartE2EDuration="1.80070676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:55:26.79923415 +0000 UTC m=+1015.579805539" watchObservedRunningTime="2026-09-15 11:55:26.80070676 +0000 UTC m=+1015.581278191"
Sep 15 11:55:26.871169 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.871136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-home\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.871352 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.871231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-tmp-dir\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.871352 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.871262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-dshm\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.871352 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.871341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-tls-certs\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.871532 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.871374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxpq\" (UniqueName: \"kubernetes.io/projected/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-kube-api-access-tmxpq\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.871532 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.871432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-model-cache\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.871643 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.871557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-home\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.871694 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.871652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-tmp-dir\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.871745 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.871727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-model-cache\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.873768 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.873737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-dshm\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.874195 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.874170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-tls-certs\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.883493 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.883469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxpq\" (UniqueName: \"kubernetes.io/projected/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-kube-api-access-tmxpq\") pod \"test-model-97d02d-kserve-54bb8d5dcc-jz8rl\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") " pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:26.979106 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:26.979072    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:27.258283 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:27.258252    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"]
Sep 15 11:55:27.305734 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:55:27.305704    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f8e7e20_1e84_4b10_9a8c_26b00dae7eda.slice/crio-aa8ad614f7728feed7a6f6f8db5fc9297525a9c1208a96172f70c69e6283ac4b WatchSource:0}: Error finding container aa8ad614f7728feed7a6f6f8db5fc9297525a9c1208a96172f70c69e6283ac4b: Status 404 returned error can't find the container with id aa8ad614f7728feed7a6f6f8db5fc9297525a9c1208a96172f70c69e6283ac4b
Sep 15 11:55:27.783208 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:27.783174    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb6b73c1-c76b-4a90-88a9-6c07780ad0fb" containerID="bdf7af673862def60dd8df3170deb4dbce7fbe93e06ade9f8ae73396b5d5655f" exitCode=0
Sep 15 11:55:27.783406 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:27.783256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd" event={"ID":"fb6b73c1-c76b-4a90-88a9-6c07780ad0fb","Type":"ContainerDied","Data":"bdf7af673862def60dd8df3170deb4dbce7fbe93e06ade9f8ae73396b5d5655f"}
Sep 15 11:55:27.784850 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:27.784824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl" event={"ID":"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda","Type":"ContainerStarted","Data":"4ef5b609ced0c4ba46793330d9a0a305ea74be7c12f06987a2fd19f9d405be7e"}
Sep 15 11:55:27.784982 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:27.784863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl" event={"ID":"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda","Type":"ContainerStarted","Data":"aa8ad614f7728feed7a6f6f8db5fc9297525a9c1208a96172f70c69e6283ac4b"}
Sep 15 11:55:27.819685 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:27.819637    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl" podStartSLOduration=1.819623524 podStartE2EDuration="1.819623524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:55:27.816577201 +0000 UTC m=+1016.597148588" watchObservedRunningTime="2026-09-15 11:55:27.819623524 +0000 UTC m=+1016.600194911"
Sep 15 11:55:28.927539 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:28.927513    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd"
Sep 15 11:55:28.991596 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:28.991567    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6b73c1-c76b-4a90-88a9-6c07780ad0fb-maas-api-service-ca\") pod \"fb6b73c1-c76b-4a90-88a9-6c07780ad0fb\" (UID: \"fb6b73c1-c76b-4a90-88a9-6c07780ad0fb\") "
Sep 15 11:55:28.991911 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:28.991862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6b73c1-c76b-4a90-88a9-6c07780ad0fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fb6b73c1-c76b-4a90-88a9-6c07780ad0fb" (UID: "fb6b73c1-c76b-4a90-88a9-6c07780ad0fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:55:29.092425 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:29.092345    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6b73c1-c76b-4a90-88a9-6c07780ad0fb-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:55:29.798392 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:29.798360    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd" event={"ID":"fb6b73c1-c76b-4a90-88a9-6c07780ad0fb","Type":"ContainerDied","Data":"2b6e9608b29dd6265c6fec1b204832762fb785ac3e9746f0a51d144e822e5973"}
Sep 15 11:55:29.798550 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:29.798394    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd"
Sep 15 11:55:29.798550 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:29.798407    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b6e9608b29dd6265c6fec1b204832762fb785ac3e9746f0a51d144e822e5973"
Sep 15 11:55:32.476689 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.476656    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"]
Sep 15 11:55:32.477195 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.476918    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp" podUID="f20da074-1db8-474d-b9fc-889a38ca5996" containerName="payload-pre-processing" containerID="cri-o://4b1a208ba70b71ed87605f27131098b91f2ae62b31a46a292fda32c34d99b9b2" gracePeriod=30
Sep 15 11:55:32.684676 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.684641    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd"]
Sep 15 11:55:32.687135 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.687107    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b407aeb7-mjjxd"]
Sep 15 11:55:32.742614 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.742565    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"
Sep 15 11:55:32.819586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.819550    2580 generic.go:356] "Generic (PLEG): container finished" podID="f20da074-1db8-474d-b9fc-889a38ca5996" containerID="4b1a208ba70b71ed87605f27131098b91f2ae62b31a46a292fda32c34d99b9b2" exitCode=0
Sep 15 11:55:32.819586 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.819589    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"
Sep 15 11:55:32.819866 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.819628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp" event={"ID":"f20da074-1db8-474d-b9fc-889a38ca5996","Type":"ContainerDied","Data":"4b1a208ba70b71ed87605f27131098b91f2ae62b31a46a292fda32c34d99b9b2"}
Sep 15 11:55:32.819866 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.819689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp" event={"ID":"f20da074-1db8-474d-b9fc-889a38ca5996","Type":"ContainerDied","Data":"2fe9cb56b456d2e7a4a4e87c765e653ff8f5aaab4b05bc373d6b1e3c6d297fe4"}
Sep 15 11:55:32.819866 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.819725    2580 scope.go:117] "RemoveContainer" containerID="4b1a208ba70b71ed87605f27131098b91f2ae62b31a46a292fda32c34d99b9b2"
Sep 15 11:55:32.827860 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.827840    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9hg9\" (UniqueName: \"kubernetes.io/projected/f20da074-1db8-474d-b9fc-889a38ca5996-kube-api-access-x9hg9\") pod \"f20da074-1db8-474d-b9fc-889a38ca5996\" (UID: \"f20da074-1db8-474d-b9fc-889a38ca5996\") "
Sep 15 11:55:32.827999 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.827886    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20da074-1db8-474d-b9fc-889a38ca5996-plugins-config-volume\") pod \"f20da074-1db8-474d-b9fc-889a38ca5996\" (UID: \"f20da074-1db8-474d-b9fc-889a38ca5996\") "
Sep 15 11:55:32.828283 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.828260    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20da074-1db8-474d-b9fc-889a38ca5996-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f20da074-1db8-474d-b9fc-889a38ca5996" (UID: "f20da074-1db8-474d-b9fc-889a38ca5996"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:55:32.830508 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.830485    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20da074-1db8-474d-b9fc-889a38ca5996-kube-api-access-x9hg9" (OuterVolumeSpecName: "kube-api-access-x9hg9") pod "f20da074-1db8-474d-b9fc-889a38ca5996" (UID: "f20da074-1db8-474d-b9fc-889a38ca5996"). InnerVolumeSpecName "kube-api-access-x9hg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:55:32.831276 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.831260    2580 scope.go:117] "RemoveContainer" containerID="4b1a208ba70b71ed87605f27131098b91f2ae62b31a46a292fda32c34d99b9b2"
Sep 15 11:55:32.831570 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:55:32.831553    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1a208ba70b71ed87605f27131098b91f2ae62b31a46a292fda32c34d99b9b2\": container with ID starting with 4b1a208ba70b71ed87605f27131098b91f2ae62b31a46a292fda32c34d99b9b2 not found: ID does not exist" containerID="4b1a208ba70b71ed87605f27131098b91f2ae62b31a46a292fda32c34d99b9b2"
Sep 15 11:55:32.831634 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.831579    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1a208ba70b71ed87605f27131098b91f2ae62b31a46a292fda32c34d99b9b2"} err="failed to get container status \"4b1a208ba70b71ed87605f27131098b91f2ae62b31a46a292fda32c34d99b9b2\": rpc error: code = NotFound desc = could not find container \"4b1a208ba70b71ed87605f27131098b91f2ae62b31a46a292fda32c34d99b9b2\": container with ID starting with 4b1a208ba70b71ed87605f27131098b91f2ae62b31a46a292fda32c34d99b9b2 not found: ID does not exist"
Sep 15 11:55:32.929176 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.929138    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9hg9\" (UniqueName: \"kubernetes.io/projected/f20da074-1db8-474d-b9fc-889a38ca5996-kube-api-access-x9hg9\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:55:32.929176 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:32.929178    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20da074-1db8-474d-b9fc-889a38ca5996-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:55:33.148663 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:33.144556    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"]
Sep 15 11:55:33.151901 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:33.151860    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b407aeb7-6d9585d8c8-5jfsp"]
Sep 15 11:55:33.731616 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:33.731586    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f20da074-1db8-474d-b9fc-889a38ca5996" path="/var/lib/kubelet/pods/f20da074-1db8-474d-b9fc-889a38ca5996/volumes"
Sep 15 11:55:33.732006 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:33.731913    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb6b73c1-c76b-4a90-88a9-6c07780ad0fb" path="/var/lib/kubelet/pods/fb6b73c1-c76b-4a90-88a9-6c07780ad0fb/volumes"
Sep 15 11:55:36.979693 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:36.979663    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:36.980073 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:36.979704    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:36.992514 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:36.992490    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:37.861438 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:37.861413    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 11:55:40.164686 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.164652    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"]
Sep 15 11:55:40.291259 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.291220    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt"]
Sep 15 11:55:40.291648 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.291634    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20da074-1db8-474d-b9fc-889a38ca5996" containerName="payload-pre-processing"
Sep 15 11:55:40.291693 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.291650    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20da074-1db8-474d-b9fc-889a38ca5996" containerName="payload-pre-processing"
Sep 15 11:55:40.291693 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.291659    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb6b73c1-c76b-4a90-88a9-6c07780ad0fb" containerName="revoke-keys"
Sep 15 11:55:40.291693 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.291664    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb6b73c1-c76b-4a90-88a9-6c07780ad0fb" containerName="revoke-keys"
Sep 15 11:55:40.291791 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.291744    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb6b73c1-c76b-4a90-88a9-6c07780ad0fb" containerName="revoke-keys"
Sep 15 11:55:40.291791 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.291754    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20da074-1db8-474d-b9fc-889a38ca5996" containerName="payload-pre-processing"
Sep 15 11:55:40.296066 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.296047    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt"
Sep 15 11:55:40.298961 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.298934    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 11:55:40.299081 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.298946    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:55:40.302717 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.302685    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt"]
Sep 15 11:55:40.402259 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.402226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5911457-bd85-4487-a5fa-fc54224ab1ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b42f08-jt7tt\" (UID: \"d5911457-bd85-4487-a5fa-fc54224ab1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt"
Sep 15 11:55:40.503139 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.503045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5911457-bd85-4487-a5fa-fc54224ab1ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b42f08-jt7tt\" (UID: \"d5911457-bd85-4487-a5fa-fc54224ab1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt"
Sep 15 11:55:40.503644 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.503623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5911457-bd85-4487-a5fa-fc54224ab1ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b42f08-jt7tt\" (UID: \"d5911457-bd85-4487-a5fa-fc54224ab1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt"
Sep 15 11:55:40.609803 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.609773    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt"
Sep 15 11:55:40.885056 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:40.885034    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt"]
Sep 15 11:55:40.886634 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:55:40.886604    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5911457_bd85_4487_a5fa_fc54224ab1ee.slice/crio-99a64dbddc6ce6d74be1fabc7624d52cff7e71fe487e5a05e0c7d842dbc19ed4 WatchSource:0}: Error finding container 99a64dbddc6ce6d74be1fabc7624d52cff7e71fe487e5a05e0c7d842dbc19ed4: Status 404 returned error can't find the container with id 99a64dbddc6ce6d74be1fabc7624d52cff7e71fe487e5a05e0c7d842dbc19ed4
Sep 15 11:55:41.876995 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:41.876906    2580 generic.go:356] "Generic (PLEG): container finished" podID="d5911457-bd85-4487-a5fa-fc54224ab1ee" containerID="18b55350ae4731856bc67e5d13a796fb7590127026d24091df3671fbf320b451" exitCode=0
Sep 15 11:55:41.876995 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:41.876956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt" event={"ID":"d5911457-bd85-4487-a5fa-fc54224ab1ee","Type":"ContainerDied","Data":"18b55350ae4731856bc67e5d13a796fb7590127026d24091df3671fbf320b451"}
Sep 15 11:55:41.877380 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:41.877000    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt" event={"ID":"d5911457-bd85-4487-a5fa-fc54224ab1ee","Type":"ContainerStarted","Data":"99a64dbddc6ce6d74be1fabc7624d52cff7e71fe487e5a05e0c7d842dbc19ed4"}
Sep 15 11:55:43.015485 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:43.015453    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt"
Sep 15 11:55:43.127712 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:43.127676    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5911457-bd85-4487-a5fa-fc54224ab1ee-maas-api-service-ca\") pod \"d5911457-bd85-4487-a5fa-fc54224ab1ee\" (UID: \"d5911457-bd85-4487-a5fa-fc54224ab1ee\") "
Sep 15 11:55:43.128148 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:43.128125    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5911457-bd85-4487-a5fa-fc54224ab1ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5911457-bd85-4487-a5fa-fc54224ab1ee" (UID: "d5911457-bd85-4487-a5fa-fc54224ab1ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:55:43.229498 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:43.229471    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5911457-bd85-4487-a5fa-fc54224ab1ee-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:55:43.889596 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:43.889569    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt"
Sep 15 11:55:43.889748 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:43.889596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt" event={"ID":"d5911457-bd85-4487-a5fa-fc54224ab1ee","Type":"ContainerDied","Data":"99a64dbddc6ce6d74be1fabc7624d52cff7e71fe487e5a05e0c7d842dbc19ed4"}
Sep 15 11:55:43.889748 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:43.889669    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99a64dbddc6ce6d74be1fabc7624d52cff7e71fe487e5a05e0c7d842dbc19ed4"
Sep 15 11:55:50.991649 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:50.991613    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"]
Sep 15 11:55:50.992142 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:50.991834    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8" podUID="6e1297b5-44fa-45f9-813a-06382923ce8d" containerName="payload-pre-processing" containerID="cri-o://d3d1fe34dc7c03ea7d9fdd3095a023ca248d7646087626050c4a3969e6a14e98" gracePeriod=30
Sep 15 11:55:51.162839 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.162680    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt"]
Sep 15 11:55:51.164348 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.164028    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b42f08-jt7tt"]
Sep 15 11:55:51.255342 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.255276    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"
Sep 15 11:55:51.405479 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.405446    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqxm8\" (UniqueName: \"kubernetes.io/projected/6e1297b5-44fa-45f9-813a-06382923ce8d-kube-api-access-tqxm8\") pod \"6e1297b5-44fa-45f9-813a-06382923ce8d\" (UID: \"6e1297b5-44fa-45f9-813a-06382923ce8d\") "
Sep 15 11:55:51.405682 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.405525    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1297b5-44fa-45f9-813a-06382923ce8d-plugins-config-volume\") pod \"6e1297b5-44fa-45f9-813a-06382923ce8d\" (UID: \"6e1297b5-44fa-45f9-813a-06382923ce8d\") "
Sep 15 11:55:51.405924 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.405898    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1297b5-44fa-45f9-813a-06382923ce8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e1297b5-44fa-45f9-813a-06382923ce8d" (UID: "6e1297b5-44fa-45f9-813a-06382923ce8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:55:51.407737 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.407703    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1297b5-44fa-45f9-813a-06382923ce8d-kube-api-access-tqxm8" (OuterVolumeSpecName: "kube-api-access-tqxm8") pod "6e1297b5-44fa-45f9-813a-06382923ce8d" (UID: "6e1297b5-44fa-45f9-813a-06382923ce8d"). InnerVolumeSpecName "kube-api-access-tqxm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:55:51.506430 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.506360    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqxm8\" (UniqueName: \"kubernetes.io/projected/6e1297b5-44fa-45f9-813a-06382923ce8d-kube-api-access-tqxm8\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:55:51.506430 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.506387    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1297b5-44fa-45f9-813a-06382923ce8d-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:55:51.731099 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.731063    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5911457-bd85-4487-a5fa-fc54224ab1ee" path="/var/lib/kubelet/pods/d5911457-bd85-4487-a5fa-fc54224ab1ee/volumes"
Sep 15 11:55:51.939355 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.939325    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e1297b5-44fa-45f9-813a-06382923ce8d" containerID="d3d1fe34dc7c03ea7d9fdd3095a023ca248d7646087626050c4a3969e6a14e98" exitCode=0
Sep 15 11:55:51.939552 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.939370    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8" event={"ID":"6e1297b5-44fa-45f9-813a-06382923ce8d","Type":"ContainerDied","Data":"d3d1fe34dc7c03ea7d9fdd3095a023ca248d7646087626050c4a3969e6a14e98"}
Sep 15 11:55:51.939552 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.939408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8" event={"ID":"6e1297b5-44fa-45f9-813a-06382923ce8d","Type":"ContainerDied","Data":"3c03b5ca5e581fbf444a9dce089d003835a119883d4a1b5a9aac66eae3186544"}
Sep 15 11:55:51.939552 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.939432    2580 scope.go:117] "RemoveContainer" containerID="d3d1fe34dc7c03ea7d9fdd3095a023ca248d7646087626050c4a3969e6a14e98"
Sep 15 11:55:51.939552 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.939371    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"
Sep 15 11:55:51.948627 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.948603    2580 scope.go:117] "RemoveContainer" containerID="d3d1fe34dc7c03ea7d9fdd3095a023ca248d7646087626050c4a3969e6a14e98"
Sep 15 11:55:51.948910 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:55:51.948891    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d1fe34dc7c03ea7d9fdd3095a023ca248d7646087626050c4a3969e6a14e98\": container with ID starting with d3d1fe34dc7c03ea7d9fdd3095a023ca248d7646087626050c4a3969e6a14e98 not found: ID does not exist" containerID="d3d1fe34dc7c03ea7d9fdd3095a023ca248d7646087626050c4a3969e6a14e98"
Sep 15 11:55:51.948976 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.948917    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d1fe34dc7c03ea7d9fdd3095a023ca248d7646087626050c4a3969e6a14e98"} err="failed to get container status \"d3d1fe34dc7c03ea7d9fdd3095a023ca248d7646087626050c4a3969e6a14e98\": rpc error: code = NotFound desc = could not find container \"d3d1fe34dc7c03ea7d9fdd3095a023ca248d7646087626050c4a3969e6a14e98\": container with ID starting with d3d1fe34dc7c03ea7d9fdd3095a023ca248d7646087626050c4a3969e6a14e98 not found: ID does not exist"
Sep 15 11:55:51.957349 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.957327    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"]
Sep 15 11:55:51.959768 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:51.959749    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b42f08-66ffdbdb98-rj7x8"]
Sep 15 11:55:53.731186 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:53.731152    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e1297b5-44fa-45f9-813a-06382923ce8d" path="/var/lib/kubelet/pods/6e1297b5-44fa-45f9-813a-06382923ce8d/volumes"
Sep 15 11:55:55.183777 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.183733    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp" podUID="83719daf-cd57-4189-b5a1-d28c4e999b8b" containerName="main" containerID="cri-o://632c566dc62862b44bef126ea349bc4e325d3a214d09240b38b141d2610e3b99" gracePeriod=45
Sep 15 11:55:55.436972 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.436912    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:55.542796 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.542768    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-model-cache\") pod \"83719daf-cd57-4189-b5a1-d28c4e999b8b\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") "
Sep 15 11:55:55.542972 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.542815    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-dshm\") pod \"83719daf-cd57-4189-b5a1-d28c4e999b8b\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") "
Sep 15 11:55:55.542972 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.542890    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-tmp-dir\") pod \"83719daf-cd57-4189-b5a1-d28c4e999b8b\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") "
Sep 15 11:55:55.543065 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.543006    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83719daf-cd57-4189-b5a1-d28c4e999b8b-tls-certs\") pod \"83719daf-cd57-4189-b5a1-d28c4e999b8b\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") "
Sep 15 11:55:55.543065 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.543035    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-home\") pod \"83719daf-cd57-4189-b5a1-d28c4e999b8b\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") "
Sep 15 11:55:55.543065 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.543051    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-model-cache" (OuterVolumeSpecName: "model-cache") pod "83719daf-cd57-4189-b5a1-d28c4e999b8b" (UID: "83719daf-cd57-4189-b5a1-d28c4e999b8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:55:55.543223 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.543083    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hls6w\" (UniqueName: \"kubernetes.io/projected/83719daf-cd57-4189-b5a1-d28c4e999b8b-kube-api-access-hls6w\") pod \"83719daf-cd57-4189-b5a1-d28c4e999b8b\" (UID: \"83719daf-cd57-4189-b5a1-d28c4e999b8b\") "
Sep 15 11:55:55.543223 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.543186    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83719daf-cd57-4189-b5a1-d28c4e999b8b" (UID: "83719daf-cd57-4189-b5a1-d28c4e999b8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:55:55.543328 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.543234    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-home" (OuterVolumeSpecName: "home") pod "83719daf-cd57-4189-b5a1-d28c4e999b8b" (UID: "83719daf-cd57-4189-b5a1-d28c4e999b8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:55:55.543428 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.543410    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:55:55.543428 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.543427    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:55:55.543547 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.543438    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:55:55.545150 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.545128    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-dshm" (OuterVolumeSpecName: "dshm") pod "83719daf-cd57-4189-b5a1-d28c4e999b8b" (UID: "83719daf-cd57-4189-b5a1-d28c4e999b8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:55:55.545251 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.545234    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83719daf-cd57-4189-b5a1-d28c4e999b8b-kube-api-access-hls6w" (OuterVolumeSpecName: "kube-api-access-hls6w") pod "83719daf-cd57-4189-b5a1-d28c4e999b8b" (UID: "83719daf-cd57-4189-b5a1-d28c4e999b8b"). InnerVolumeSpecName "kube-api-access-hls6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:55:55.545298 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.545241    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83719daf-cd57-4189-b5a1-d28c4e999b8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83719daf-cd57-4189-b5a1-d28c4e999b8b" (UID: "83719daf-cd57-4189-b5a1-d28c4e999b8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:55:55.644559 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.644523    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83719daf-cd57-4189-b5a1-d28c4e999b8b-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:55:55.644559 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.644558    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hls6w\" (UniqueName: \"kubernetes.io/projected/83719daf-cd57-4189-b5a1-d28c4e999b8b-kube-api-access-hls6w\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:55:55.644559 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.644567    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83719daf-cd57-4189-b5a1-d28c4e999b8b-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:55:55.966229 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.966193    2580 generic.go:356] "Generic (PLEG): container finished" podID="83719daf-cd57-4189-b5a1-d28c4e999b8b" containerID="632c566dc62862b44bef126ea349bc4e325d3a214d09240b38b141d2610e3b99" exitCode=0
Sep 15 11:55:55.966388 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.966239    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"
Sep 15 11:55:55.966388 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.966273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp" event={"ID":"83719daf-cd57-4189-b5a1-d28c4e999b8b","Type":"ContainerDied","Data":"632c566dc62862b44bef126ea349bc4e325d3a214d09240b38b141d2610e3b99"}
Sep 15 11:55:55.966388 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.966316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp" event={"ID":"83719daf-cd57-4189-b5a1-d28c4e999b8b","Type":"ContainerDied","Data":"cb731508b939c9836ad496dfaf740317f8f547782f0012b79efba96810ce784e"}
Sep 15 11:55:55.966388 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.966334    2580 scope.go:117] "RemoveContainer" containerID="632c566dc62862b44bef126ea349bc4e325d3a214d09240b38b141d2610e3b99"
Sep 15 11:55:55.975757 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.975741    2580 scope.go:117] "RemoveContainer" containerID="632c566dc62862b44bef126ea349bc4e325d3a214d09240b38b141d2610e3b99"
Sep 15 11:55:55.976054 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:55:55.976034    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632c566dc62862b44bef126ea349bc4e325d3a214d09240b38b141d2610e3b99\": container with ID starting with 632c566dc62862b44bef126ea349bc4e325d3a214d09240b38b141d2610e3b99 not found: ID does not exist" containerID="632c566dc62862b44bef126ea349bc4e325d3a214d09240b38b141d2610e3b99"
Sep 15 11:55:55.976118 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.976061    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632c566dc62862b44bef126ea349bc4e325d3a214d09240b38b141d2610e3b99"} err="failed to get container status \"632c566dc62862b44bef126ea349bc4e325d3a214d09240b38b141d2610e3b99\": rpc error: code = NotFound desc = could not find container \"632c566dc62862b44bef126ea349bc4e325d3a214d09240b38b141d2610e3b99\": container with ID starting with 632c566dc62862b44bef126ea349bc4e325d3a214d09240b38b141d2610e3b99 not found: ID does not exist"
Sep 15 11:55:55.985651 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.985629    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"]
Sep 15 11:55:55.989620 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:55.989572    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-b42f08/autores-model-b42f08-kserve-7fc9bd665f-gqtsp"]
Sep 15 11:55:57.730961 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:57.730928    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83719daf-cd57-4189-b5a1-d28c4e999b8b" path="/var/lib/kubelet/pods/83719daf-cd57-4189-b5a1-d28c4e999b8b/volumes"
Sep 15 11:55:58.080003 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:58.079909    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"]
Sep 15 11:55:58.080219 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:58.080173    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l" podUID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerName="istio-proxy" containerID="cri-o://3dfe43f96e1db9d388689f87edf6b796f969519e7957c4d3949fcfded71212dd" gracePeriod=30
Sep 15 11:55:58.975788 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:58.975753    2580 patch_prober.go:28] interesting pod/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 11:55:58.976226 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:55:58.975834    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l" podUID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 11:56:03.338126 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.338101    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:56:03.520213 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520183    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-token\") pod \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") "
Sep 15 11:56:03.520389 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520235    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istiod-ca-cert\") pod \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") "
Sep 15 11:56:03.520389 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520265    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-data\") pod \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") "
Sep 15 11:56:03.520389 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520309    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-envoy\") pod \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") "
Sep 15 11:56:03.520389 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-workload-certs\") pod \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") "
Sep 15 11:56:03.520601 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520393    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-podinfo\") pod \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") "
Sep 15 11:56:03.520601 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520438    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-credential-socket\") pod \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") "
Sep 15 11:56:03.520601 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520488    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gnpz\" (UniqueName: \"kubernetes.io/projected/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-kube-api-access-8gnpz\") pod \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") "
Sep 15 11:56:03.520601 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520537    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-workload-socket\") pod \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\" (UID: \"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83\") "
Sep 15 11:56:03.520813 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520599    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-data" (OuterVolumeSpecName: "istio-data") pod "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" (UID: "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:56:03.520813 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520742    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" (UID: "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:56:03.520813 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520741    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" (UID: "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:56:03.521011 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520898    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istiod-ca-cert\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:56:03.521011 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520914    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-data\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:56:03.521011 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.520928    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-workload-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:56:03.521174 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.521018    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" (UID: "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:56:03.521174 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.521030    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" (UID: "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:56:03.522555 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.522531    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-token" (OuterVolumeSpecName: "istio-token") pod "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" (UID: "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:56:03.522555 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.522535    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" (UID: "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:56:03.522724 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.522597    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-kube-api-access-8gnpz" (OuterVolumeSpecName: "kube-api-access-8gnpz") pod "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" (UID: "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83"). InnerVolumeSpecName "kube-api-access-8gnpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:56:03.522803 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.522779    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" (UID: "d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 11:56:03.621373 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.621347    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gnpz\" (UniqueName: \"kubernetes.io/projected/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-kube-api-access-8gnpz\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:56:03.621373 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.621372    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-workload-socket\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:56:03.621529 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.621383    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-token\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:56:03.621529 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.621392    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-envoy\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:56:03.621529 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.621399    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-istio-podinfo\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:56:03.621529 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:03.621407    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83-credential-socket\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:56:04.021901 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:04.021849    2580 generic.go:356] "Generic (PLEG): container finished" podID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerID="3dfe43f96e1db9d388689f87edf6b796f969519e7957c4d3949fcfded71212dd" exitCode=0
Sep 15 11:56:04.022096 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:04.021911    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"
Sep 15 11:56:04.022096 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:04.021934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l" event={"ID":"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83","Type":"ContainerDied","Data":"3dfe43f96e1db9d388689f87edf6b796f969519e7957c4d3949fcfded71212dd"}
Sep 15 11:56:04.022096 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:04.021985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l" event={"ID":"d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83","Type":"ContainerDied","Data":"2dc603eca23c06104a7d685882d61ad5aa807f88375866d352793373190c2b78"}
Sep 15 11:56:04.022096 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:04.022008    2580 scope.go:117] "RemoveContainer" containerID="3dfe43f96e1db9d388689f87edf6b796f969519e7957c4d3949fcfded71212dd"
Sep 15 11:56:04.031757 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:04.031742    2580 scope.go:117] "RemoveContainer" containerID="2f90aa3b743ac160230dd433f51e3e34ab7d8cec92fc142dd69f59da084e43a9"
Sep 15 11:56:04.040493 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:04.040476    2580 scope.go:117] "RemoveContainer" containerID="3dfe43f96e1db9d388689f87edf6b796f969519e7957c4d3949fcfded71212dd"
Sep 15 11:56:04.040602 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:04.040580    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"]
Sep 15 11:56:04.040769 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:56:04.040750    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dfe43f96e1db9d388689f87edf6b796f969519e7957c4d3949fcfded71212dd\": container with ID starting with 3dfe43f96e1db9d388689f87edf6b796f969519e7957c4d3949fcfded71212dd not found: ID does not exist" containerID="3dfe43f96e1db9d388689f87edf6b796f969519e7957c4d3949fcfded71212dd"
Sep 15 11:56:04.040822 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:04.040776    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfe43f96e1db9d388689f87edf6b796f969519e7957c4d3949fcfded71212dd"} err="failed to get container status \"3dfe43f96e1db9d388689f87edf6b796f969519e7957c4d3949fcfded71212dd\": rpc error: code = NotFound desc = could not find container \"3dfe43f96e1db9d388689f87edf6b796f969519e7957c4d3949fcfded71212dd\": container with ID starting with 3dfe43f96e1db9d388689f87edf6b796f969519e7957c4d3949fcfded71212dd not found: ID does not exist"
Sep 15 11:56:04.040822 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:04.040793    2580 scope.go:117] "RemoveContainer" containerID="2f90aa3b743ac160230dd433f51e3e34ab7d8cec92fc142dd69f59da084e43a9"
Sep 15 11:56:04.041053 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:56:04.041038    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f90aa3b743ac160230dd433f51e3e34ab7d8cec92fc142dd69f59da084e43a9\": container with ID starting with 2f90aa3b743ac160230dd433f51e3e34ab7d8cec92fc142dd69f59da084e43a9 not found: ID does not exist" containerID="2f90aa3b743ac160230dd433f51e3e34ab7d8cec92fc142dd69f59da084e43a9"
Sep 15 11:56:04.041096 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:04.041058    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f90aa3b743ac160230dd433f51e3e34ab7d8cec92fc142dd69f59da084e43a9"} err="failed to get container status \"2f90aa3b743ac160230dd433f51e3e34ab7d8cec92fc142dd69f59da084e43a9\": rpc error: code = NotFound desc = could not find container \"2f90aa3b743ac160230dd433f51e3e34ab7d8cec92fc142dd69f59da084e43a9\": container with ID starting with 2f90aa3b743ac160230dd433f51e3e34ab7d8cec92fc142dd69f59da084e43a9 not found: ID does not exist"
Sep 15 11:56:04.044750 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:04.044730    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-b42f08-openshift-default-8f44cc586-skc9l"]
Sep 15 11:56:05.733133 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.733093    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" path="/var/lib/kubelet/pods/d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83/volumes"
Sep 15 11:56:05.817320 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.817285    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"]
Sep 15 11:56:05.817748 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.817730    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1297b5-44fa-45f9-813a-06382923ce8d" containerName="payload-pre-processing"
Sep 15 11:56:05.817840 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.817750    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1297b5-44fa-45f9-813a-06382923ce8d" containerName="payload-pre-processing"
Sep 15 11:56:05.817840 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.817786    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerName="istio-proxy"
Sep 15 11:56:05.817840 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.817797    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerName="istio-proxy"
Sep 15 11:56:05.817840 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.817808    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83719daf-cd57-4189-b5a1-d28c4e999b8b" containerName="main"
Sep 15 11:56:05.817840 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.817817    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="83719daf-cd57-4189-b5a1-d28c4e999b8b" containerName="main"
Sep 15 11:56:05.817840 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.817834    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5911457-bd85-4487-a5fa-fc54224ab1ee" containerName="revoke-keys"
Sep 15 11:56:05.818172 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.817842    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5911457-bd85-4487-a5fa-fc54224ab1ee" containerName="revoke-keys"
Sep 15 11:56:05.818172 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.817959    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerName="istio-proxy"
Sep 15 11:56:05.818172 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.817975    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e1297b5-44fa-45f9-813a-06382923ce8d" containerName="payload-pre-processing"
Sep 15 11:56:05.818172 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.817988    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="83719daf-cd57-4189-b5a1-d28c4e999b8b" containerName="main"
Sep 15 11:56:05.818172 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.817996    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5911457-bd85-4487-a5fa-fc54224ab1ee" containerName="revoke-keys"
Sep 15 11:56:05.818172 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.818006    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerName="istio-proxy"
Sep 15 11:56:05.822796 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.822773    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"
Sep 15 11:56:05.825139 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.825115    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-2ff8fe-dockercfg-9f6sh\""
Sep 15 11:56:05.825250 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.825208    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-2ff8fe\""
Sep 15 11:56:05.831520 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.831495    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"]
Sep 15 11:56:05.941410 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.941372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhs26\" (UniqueName: \"kubernetes.io/projected/15e0263b-66ce-4f05-9799-8c257bcbd8be-kube-api-access-dhs26\") pod \"payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8\" (UID: \"15e0263b-66ce-4f05-9799-8c257bcbd8be\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"
Sep 15 11:56:05.941579 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:05.941420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e0263b-66ce-4f05-9799-8c257bcbd8be-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8\" (UID: \"15e0263b-66ce-4f05-9799-8c257bcbd8be\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"
Sep 15 11:56:06.042243 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:06.042156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhs26\" (UniqueName: \"kubernetes.io/projected/15e0263b-66ce-4f05-9799-8c257bcbd8be-kube-api-access-dhs26\") pod \"payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8\" (UID: \"15e0263b-66ce-4f05-9799-8c257bcbd8be\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"
Sep 15 11:56:06.042243 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:06.042225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e0263b-66ce-4f05-9799-8c257bcbd8be-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8\" (UID: \"15e0263b-66ce-4f05-9799-8c257bcbd8be\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"
Sep 15 11:56:06.042978 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:06.042954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e0263b-66ce-4f05-9799-8c257bcbd8be-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8\" (UID: \"15e0263b-66ce-4f05-9799-8c257bcbd8be\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"
Sep 15 11:56:06.050370 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:06.050345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhs26\" (UniqueName: \"kubernetes.io/projected/15e0263b-66ce-4f05-9799-8c257bcbd8be-kube-api-access-dhs26\") pod \"payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8\" (UID: \"15e0263b-66ce-4f05-9799-8c257bcbd8be\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"
Sep 15 11:56:06.134498 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:06.134461    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"
Sep 15 11:56:06.405781 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:06.405750    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"]
Sep 15 11:56:06.407328 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:56:06.407300    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e0263b_66ce_4f05_9799_8c257bcbd8be.slice/crio-ae587bb767fe4eff2ea390d995dd392c19e81c1f60b7bb9e652f2c4e57a2c0d0 WatchSource:0}: Error finding container ae587bb767fe4eff2ea390d995dd392c19e81c1f60b7bb9e652f2c4e57a2c0d0: Status 404 returned error can't find the container with id ae587bb767fe4eff2ea390d995dd392c19e81c1f60b7bb9e652f2c4e57a2c0d0
Sep 15 11:56:07.048959 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:07.048924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8" event={"ID":"15e0263b-66ce-4f05-9799-8c257bcbd8be","Type":"ContainerStarted","Data":"c0239b5d4da420c21ee89aafc3f3d3ba1e2bd33531ac7945297e1bcdef2a163f"}
Sep 15 11:56:07.049429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:07.048986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8" event={"ID":"15e0263b-66ce-4f05-9799-8c257bcbd8be","Type":"ContainerStarted","Data":"ae587bb767fe4eff2ea390d995dd392c19e81c1f60b7bb9e652f2c4e57a2c0d0"}
Sep 15 11:56:07.049429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:07.049123    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"
Sep 15 11:56:07.068993 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:07.068936    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8" podStartSLOduration=2.068920929 podStartE2EDuration="2.068920929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:07.06633169 +0000 UTC m=+1055.846903081" watchObservedRunningTime="2026-09-15 11:56:07.068920929 +0000 UTC m=+1055.849492316"
Sep 15 11:56:18.057606 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:18.057574    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"
Sep 15 11:56:47.095849 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.095809    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"]
Sep 15 11:56:47.096409 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.096395    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerName="istio-proxy"
Sep 15 11:56:47.096487 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.096416    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1af3bc7-bfa4-49b3-8ef4-1a9d2ddcff83" containerName="istio-proxy"
Sep 15 11:56:47.101238 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.101216    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.104576 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.104555    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-29d630\"/\"premium-w0-29d630-kserve-self-signed-certs\""
Sep 15 11:56:47.113985 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.113965    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"]
Sep 15 11:56:47.217843 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.217811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-tls-certs\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.217991 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.217858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-model-cache\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.217991 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.217926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-tmp-dir\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.217991 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.217985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-home\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.218125 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.218005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vb6h\" (UniqueName: \"kubernetes.io/projected/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-kube-api-access-9vb6h\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.218125 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.218063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-dshm\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.318728 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.318678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-tls-certs\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.318973 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.318746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-model-cache\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.318973 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.318764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-tmp-dir\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.318973 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.318793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-home\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.318973 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.318812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vb6h\" (UniqueName: \"kubernetes.io/projected/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-kube-api-access-9vb6h\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.318973 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.318841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-dshm\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.319275 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.319244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-tmp-dir\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.319393 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.319291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-home\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.319393 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.319321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-model-cache\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.321045 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.321025    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-dshm\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.321285 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.321267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-tls-certs\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.331519 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.331500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vb6h\" (UniqueName: \"kubernetes.io/projected/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-kube-api-access-9vb6h\") pod \"premium-w0-29d630-kserve-d4f876c56-xqbzl\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") " pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.410176 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.410105    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:47.688153 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:47.688124    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"]
Sep 15 11:56:47.689126 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:56:47.689090    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cea3d4a_e390_4dcb_a8a9_3c34d8005fbe.slice/crio-c8177364c67a3f43a8b65f2804f83e6cdbdd4893c38a3abd06a37384d8a8f790 WatchSource:0}: Error finding container c8177364c67a3f43a8b65f2804f83e6cdbdd4893c38a3abd06a37384d8a8f790: Status 404 returned error can't find the container with id c8177364c67a3f43a8b65f2804f83e6cdbdd4893c38a3abd06a37384d8a8f790
Sep 15 11:56:48.288473 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:48.288431    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl" event={"ID":"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe","Type":"ContainerStarted","Data":"425ceef44763354af4884b0662449a84f46fa7b837a1b0a394028c5694601fcb"}
Sep 15 11:56:48.288906 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:48.288485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl" event={"ID":"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe","Type":"ContainerStarted","Data":"c8177364c67a3f43a8b65f2804f83e6cdbdd4893c38a3abd06a37384d8a8f790"}
Sep 15 11:56:48.308596 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:48.308553    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl" podStartSLOduration=1.308537509 podStartE2EDuration="1.308537509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:48.308372617 +0000 UTC m=+1097.088944003" watchObservedRunningTime="2026-09-15 11:56:48.308537509 +0000 UTC m=+1097.089108899"
Sep 15 11:56:54.554361 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.554328    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"]
Sep 15 11:56:54.557805 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.557787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.560027 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.560001    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac76e5\"/\"premium-w4-ac76e5-kserve-self-signed-certs\""
Sep 15 11:56:54.566619 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.566595    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"]
Sep 15 11:56:54.582820 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.582795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-home\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.582955 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.582826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-dshm\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.582955 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.582842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbplw\" (UniqueName: \"kubernetes.io/projected/40091be4-fed1-4e6b-bded-82882eda3801-kube-api-access-wbplw\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.583027 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.582980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-model-cache\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.583067 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.583020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-tmp-dir\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.583102 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.583067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40091be4-fed1-4e6b-bded-82882eda3801-tls-certs\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.684133 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.684098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-home\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.684133 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.684133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-dshm\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.684329 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.684152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbplw\" (UniqueName: \"kubernetes.io/projected/40091be4-fed1-4e6b-bded-82882eda3801-kube-api-access-wbplw\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.684329 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.684222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-model-cache\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.684329 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.684250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-tmp-dir\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.684329 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.684289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40091be4-fed1-4e6b-bded-82882eda3801-tls-certs\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.684552 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.684537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-home\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.684615 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.684593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-model-cache\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.684693 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.684670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-tmp-dir\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.686371 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.686351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-dshm\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.686664 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.686649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40091be4-fed1-4e6b-bded-82882eda3801-tls-certs\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.692410 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.692386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbplw\" (UniqueName: \"kubernetes.io/projected/40091be4-fed1-4e6b-bded-82882eda3801-kube-api-access-wbplw\") pod \"premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") " pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:54.868689 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:54.868604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:56:55.141786 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:55.141759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"]
Sep 15 11:56:55.143894 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:56:55.143842    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40091be4_fed1_4e6b_bded_82882eda3801.slice/crio-139a79dfec19dfd2eededfc0d044b9d8ab7ce35a025b7a1bdde87d6d3336141a WatchSource:0}: Error finding container 139a79dfec19dfd2eededfc0d044b9d8ab7ce35a025b7a1bdde87d6d3336141a: Status 404 returned error can't find the container with id 139a79dfec19dfd2eededfc0d044b9d8ab7ce35a025b7a1bdde87d6d3336141a
Sep 15 11:56:55.331685 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:55.331653    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn" event={"ID":"40091be4-fed1-4e6b-bded-82882eda3801","Type":"ContainerStarted","Data":"d5b8167a0468026b170a6388e4d707ca2864083ecb1171ecf0b6907950980f0b"}
Sep 15 11:56:55.331854 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:55.331697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn" event={"ID":"40091be4-fed1-4e6b-bded-82882eda3801","Type":"ContainerStarted","Data":"139a79dfec19dfd2eededfc0d044b9d8ab7ce35a025b7a1bdde87d6d3336141a"}
Sep 15 11:56:55.350649 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:55.350602    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn" podStartSLOduration=1.350589794 podStartE2EDuration="1.350589794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:55.348829807 +0000 UTC m=+1104.129401197" watchObservedRunningTime="2026-09-15 11:56:55.350589794 +0000 UTC m=+1104.131161182"
Sep 15 11:56:57.410280 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:57.410244    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:57.410639 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:57.410299    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:57.422971 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:57.422935    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:56:58.359617 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:56:58.359586    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 11:57:03.847261 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:03.847230    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"]
Sep 15 11:57:03.850746 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:03.850723    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:03.853141 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:03.853119    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-2ff8fe\"/\"explicit-model-2ff8fe-kserve-self-signed-certs\""
Sep 15 11:57:03.854144 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:03.854123    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-2ff8fe\"/\"openshift-service-ca.crt\""
Sep 15 11:57:03.854243 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:03.854228    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-2ff8fe\"/\"default-dockercfg-nvfjx\""
Sep 15 11:57:03.854305 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:03.854228    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-2ff8fe\"/\"kube-root-ca.crt\""
Sep 15 11:57:03.861706 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:03.861686    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"]
Sep 15 11:57:03.970578 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:03.970548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlpg4\" (UniqueName: \"kubernetes.io/projected/86c40c3e-f126-4d73-8988-b0b92977d4c9-kube-api-access-nlpg4\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:03.970731 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:03.970602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-dshm\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:03.970731 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:03.970645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-model-cache\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:03.970817 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:03.970723    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-tmp-dir\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:03.970817 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:03.970759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86c40c3e-f126-4d73-8988-b0b92977d4c9-tls-certs\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:03.970907 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:03.970809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-home\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:04.072278 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.072240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-tmp-dir\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:04.072455 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.072287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86c40c3e-f126-4d73-8988-b0b92977d4c9-tls-certs\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:04.072455 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.072338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-home\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:04.072455 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.072438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlpg4\" (UniqueName: \"kubernetes.io/projected/86c40c3e-f126-4d73-8988-b0b92977d4c9-kube-api-access-nlpg4\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:04.072597 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.072518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-dshm\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:04.072597 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.072551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-model-cache\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:04.072713 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.072691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-tmp-dir\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:04.072856 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.072833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-home\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:04.072963 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.072911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-model-cache\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:04.074814 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.074790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-dshm\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:04.074943 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.074926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86c40c3e-f126-4d73-8988-b0b92977d4c9-tls-certs\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:04.080676 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.080657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlpg4\" (UniqueName: \"kubernetes.io/projected/86c40c3e-f126-4d73-8988-b0b92977d4c9-kube-api-access-nlpg4\") pod \"explicit-model-2ff8fe-kserve-648d68fc59-96b5b\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") " pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:04.161860 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.161770    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:04.443282 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.443252    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"]
Sep 15 11:57:04.444396 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:57:04.444362    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86c40c3e_f126_4d73_8988_b0b92977d4c9.slice/crio-8a8a591f506c571959533f871d987bdfb0e68597e3c3b2bc3426ba124164c613 WatchSource:0}: Error finding container 8a8a591f506c571959533f871d987bdfb0e68597e3c3b2bc3426ba124164c613: Status 404 returned error can't find the container with id 8a8a591f506c571959533f871d987bdfb0e68597e3c3b2bc3426ba124164c613
Sep 15 11:57:04.869060 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.868974    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:57:04.869060 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.869022    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:57:04.881593 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:04.881564    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:57:05.389648 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:05.389612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b" event={"ID":"86c40c3e-f126-4d73-8988-b0b92977d4c9","Type":"ContainerStarted","Data":"7b31b71bcdf6044b59821a2cb6b4980630a73f37b202a5f0143ccb3a1247e214"}
Sep 15 11:57:05.389836 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:05.389661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b" event={"ID":"86c40c3e-f126-4d73-8988-b0b92977d4c9","Type":"ContainerStarted","Data":"8a8a591f506c571959533f871d987bdfb0e68597e3c3b2bc3426ba124164c613"}
Sep 15 11:57:05.403108 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:05.403083    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 11:57:05.409843 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:05.409800    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b" podStartSLOduration=2.4097867649999998 podStartE2EDuration="2.409786765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:57:05.407488679 +0000 UTC m=+1114.188060068" watchObservedRunningTime="2026-09-15 11:57:05.409786765 +0000 UTC m=+1114.190358153"
Sep 15 11:57:14.162345 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:14.162310    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:14.162736 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:14.162366    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:14.174353 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:14.174324    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:14.452308 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:14.452229    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:27.681471 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.681431    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"]
Sep 15 11:57:27.685541 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.685517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.688506 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.688481    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e9bc87\"/\"kube-root-ca.crt\""
Sep 15 11:57:27.689195 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.689177    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e9bc87\"/\"default-dockercfg-jm9xl\""
Sep 15 11:57:27.689340 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.689323    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e9bc87\"/\"openshift-service-ca.crt\""
Sep 15 11:57:27.689420 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.689372    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e9bc87\"/\"test-model-e9bc87-kserve-self-signed-certs\""
Sep 15 11:57:27.698629 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.698606    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"]
Sep 15 11:57:27.802135 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.802108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-model-cache\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.802305 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.802148    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvq45\" (UniqueName: \"kubernetes.io/projected/a3273265-dd2e-420f-af85-e1bcea64eed6-kube-api-access-tvq45\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.802305 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.802244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-home\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.802398 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.802311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-tmp-dir\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.802398 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.802370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3273265-dd2e-420f-af85-e1bcea64eed6-tls-certs\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.802474 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.802418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-dshm\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.903806 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.903761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvq45\" (UniqueName: \"kubernetes.io/projected/a3273265-dd2e-420f-af85-e1bcea64eed6-kube-api-access-tvq45\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.904020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.903828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-home\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.904020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.903881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-tmp-dir\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.904020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.903904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3273265-dd2e-420f-af85-e1bcea64eed6-tls-certs\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.904020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.903995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-dshm\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.904234 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.904062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-model-cache\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.904333 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.904310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-home\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.904400 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.904336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-tmp-dir\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.904441 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.904410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-model-cache\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.906258 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.906235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-dshm\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.906547 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.906531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3273265-dd2e-420f-af85-e1bcea64eed6-tls-certs\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.916214 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.916182    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvq45\" (UniqueName: \"kubernetes.io/projected/a3273265-dd2e-420f-af85-e1bcea64eed6-kube-api-access-tvq45\") pod \"test-model-e9bc87-kserve-75c4b495f7-5jkt5\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") " pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:27.996009 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:27.995915    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:28.280108 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:28.280080    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"]
Sep 15 11:57:28.281625 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:57:28.281598    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3273265_dd2e_420f_af85_e1bcea64eed6.slice/crio-6bee6aa65804394f2fa0dd8f63ced71faf0580e74351a5cba019659444705f72 WatchSource:0}: Error finding container 6bee6aa65804394f2fa0dd8f63ced71faf0580e74351a5cba019659444705f72: Status 404 returned error can't find the container with id 6bee6aa65804394f2fa0dd8f63ced71faf0580e74351a5cba019659444705f72
Sep 15 11:57:28.524313 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:28.524279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5" event={"ID":"a3273265-dd2e-420f-af85-e1bcea64eed6","Type":"ContainerStarted","Data":"8a54eac684cdc8e6b263a9d4cc0ad25c5e1d8f33d819baa8cf025dfc3442e8ef"}
Sep 15 11:57:28.524490 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:28.524326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5" event={"ID":"a3273265-dd2e-420f-af85-e1bcea64eed6","Type":"ContainerStarted","Data":"6bee6aa65804394f2fa0dd8f63ced71faf0580e74351a5cba019659444705f72"}
Sep 15 11:57:28.545560 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:28.545457    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5" podStartSLOduration=1.545438324 podStartE2EDuration="1.545438324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:57:28.544929094 +0000 UTC m=+1137.325500483" watchObservedRunningTime="2026-09-15 11:57:28.545438324 +0000 UTC m=+1137.326009713"
Sep 15 11:57:37.996247 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:37.996212    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:37.996639 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:37.996295    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:38.008915 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:38.008866    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:38.594620 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:38.594593    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 11:57:39.285434 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:39.285399    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2"]
Sep 15 11:57:39.289180 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:39.289160    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2"
Sep 15 11:57:39.291503 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:39.291476    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 11:57:39.291503 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:39.291476    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:57:39.296028 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:39.296003    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2"]
Sep 15 11:57:39.314254 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:39.314225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703d43bb-0028-4c42-a0dc-880819b52a3d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2\" (UID: \"703d43bb-0028-4c42-a0dc-880819b52a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2"
Sep 15 11:57:39.415571 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:39.415541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703d43bb-0028-4c42-a0dc-880819b52a3d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2\" (UID: \"703d43bb-0028-4c42-a0dc-880819b52a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2"
Sep 15 11:57:39.416175 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:39.416155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703d43bb-0028-4c42-a0dc-880819b52a3d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2\" (UID: \"703d43bb-0028-4c42-a0dc-880819b52a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2"
Sep 15 11:57:39.602571 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:39.602492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2"
Sep 15 11:57:39.875254 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:39.875225    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2"]
Sep 15 11:57:39.876261 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:57:39.876233    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703d43bb_0028_4c42_a0dc_880819b52a3d.slice/crio-839346d50ec7864eefae8bb343ad6252087bb5762ce0c57d3e82483364598a5a WatchSource:0}: Error finding container 839346d50ec7864eefae8bb343ad6252087bb5762ce0c57d3e82483364598a5a: Status 404 returned error can't find the container with id 839346d50ec7864eefae8bb343ad6252087bb5762ce0c57d3e82483364598a5a
Sep 15 11:57:40.596479 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:40.596446    2580 generic.go:356] "Generic (PLEG): container finished" podID="703d43bb-0028-4c42-a0dc-880819b52a3d" containerID="360c8239ad91e835bc74912f1bd270a79abaa0aa760123a7460529d2334538bf" exitCode=0
Sep 15 11:57:40.596842 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:40.596530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2" event={"ID":"703d43bb-0028-4c42-a0dc-880819b52a3d","Type":"ContainerDied","Data":"360c8239ad91e835bc74912f1bd270a79abaa0aa760123a7460529d2334538bf"}
Sep 15 11:57:40.596842 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:40.596632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2" event={"ID":"703d43bb-0028-4c42-a0dc-880819b52a3d","Type":"ContainerStarted","Data":"839346d50ec7864eefae8bb343ad6252087bb5762ce0c57d3e82483364598a5a"}
Sep 15 11:57:41.737427 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:41.737405    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2"
Sep 15 11:57:41.837644 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:41.837606    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703d43bb-0028-4c42-a0dc-880819b52a3d-maas-api-service-ca\") pod \"703d43bb-0028-4c42-a0dc-880819b52a3d\" (UID: \"703d43bb-0028-4c42-a0dc-880819b52a3d\") "
Sep 15 11:57:41.838009 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:41.837981    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703d43bb-0028-4c42-a0dc-880819b52a3d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "703d43bb-0028-4c42-a0dc-880819b52a3d" (UID: "703d43bb-0028-4c42-a0dc-880819b52a3d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:57:41.938549 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:41.938505    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703d43bb-0028-4c42-a0dc-880819b52a3d-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:57:42.609749 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:42.609717    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2"
Sep 15 11:57:42.609950 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:42.609752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2" event={"ID":"703d43bb-0028-4c42-a0dc-880819b52a3d","Type":"ContainerDied","Data":"839346d50ec7864eefae8bb343ad6252087bb5762ce0c57d3e82483364598a5a"}
Sep 15 11:57:42.609950 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:42.609797    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="839346d50ec7864eefae8bb343ad6252087bb5762ce0c57d3e82483364598a5a"
Sep 15 11:57:43.648543 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:43.648506    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"]
Sep 15 11:57:50.158931 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.158894    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"]
Sep 15 11:57:50.159411 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.159172    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8" podUID="15e0263b-66ce-4f05-9799-8c257bcbd8be" containerName="payload-pre-processing" containerID="cri-o://c0239b5d4da420c21ee89aafc3f3d3ba1e2bd33531ac7945297e1bcdef2a163f" gracePeriod=30
Sep 15 11:57:50.344672 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.344629    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2"]
Sep 15 11:57:50.347558 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.347529    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2ff8fe-vk2m2"]
Sep 15 11:57:50.404445 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.404421    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"
Sep 15 11:57:50.521865 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.521836    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e0263b-66ce-4f05-9799-8c257bcbd8be-plugins-config-volume\") pod \"15e0263b-66ce-4f05-9799-8c257bcbd8be\" (UID: \"15e0263b-66ce-4f05-9799-8c257bcbd8be\") "
Sep 15 11:57:50.522057 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.521965    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhs26\" (UniqueName: \"kubernetes.io/projected/15e0263b-66ce-4f05-9799-8c257bcbd8be-kube-api-access-dhs26\") pod \"15e0263b-66ce-4f05-9799-8c257bcbd8be\" (UID: \"15e0263b-66ce-4f05-9799-8c257bcbd8be\") "
Sep 15 11:57:50.522200 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.522178    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e0263b-66ce-4f05-9799-8c257bcbd8be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15e0263b-66ce-4f05-9799-8c257bcbd8be" (UID: "15e0263b-66ce-4f05-9799-8c257bcbd8be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:57:50.524092 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.524066    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e0263b-66ce-4f05-9799-8c257bcbd8be-kube-api-access-dhs26" (OuterVolumeSpecName: "kube-api-access-dhs26") pod "15e0263b-66ce-4f05-9799-8c257bcbd8be" (UID: "15e0263b-66ce-4f05-9799-8c257bcbd8be"). InnerVolumeSpecName "kube-api-access-dhs26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:57:50.623483 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.623448    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e0263b-66ce-4f05-9799-8c257bcbd8be-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:57:50.623483 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.623480    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhs26\" (UniqueName: \"kubernetes.io/projected/15e0263b-66ce-4f05-9799-8c257bcbd8be-kube-api-access-dhs26\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:57:50.659071 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.659038    2580 generic.go:356] "Generic (PLEG): container finished" podID="15e0263b-66ce-4f05-9799-8c257bcbd8be" containerID="c0239b5d4da420c21ee89aafc3f3d3ba1e2bd33531ac7945297e1bcdef2a163f" exitCode=0
Sep 15 11:57:50.659255 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.659095    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"
Sep 15 11:57:50.659255 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.659119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8" event={"ID":"15e0263b-66ce-4f05-9799-8c257bcbd8be","Type":"ContainerDied","Data":"c0239b5d4da420c21ee89aafc3f3d3ba1e2bd33531ac7945297e1bcdef2a163f"}
Sep 15 11:57:50.659255 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.659169    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8" event={"ID":"15e0263b-66ce-4f05-9799-8c257bcbd8be","Type":"ContainerDied","Data":"ae587bb767fe4eff2ea390d995dd392c19e81c1f60b7bb9e652f2c4e57a2c0d0"}
Sep 15 11:57:50.659255 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.659191    2580 scope.go:117] "RemoveContainer" containerID="c0239b5d4da420c21ee89aafc3f3d3ba1e2bd33531ac7945297e1bcdef2a163f"
Sep 15 11:57:50.669474 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.669459    2580 scope.go:117] "RemoveContainer" containerID="c0239b5d4da420c21ee89aafc3f3d3ba1e2bd33531ac7945297e1bcdef2a163f"
Sep 15 11:57:50.669764 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:57:50.669743    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0239b5d4da420c21ee89aafc3f3d3ba1e2bd33531ac7945297e1bcdef2a163f\": container with ID starting with c0239b5d4da420c21ee89aafc3f3d3ba1e2bd33531ac7945297e1bcdef2a163f not found: ID does not exist" containerID="c0239b5d4da420c21ee89aafc3f3d3ba1e2bd33531ac7945297e1bcdef2a163f"
Sep 15 11:57:50.669816 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.669772    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0239b5d4da420c21ee89aafc3f3d3ba1e2bd33531ac7945297e1bcdef2a163f"} err="failed to get container status \"c0239b5d4da420c21ee89aafc3f3d3ba1e2bd33531ac7945297e1bcdef2a163f\": rpc error: code = NotFound desc = could not find container \"c0239b5d4da420c21ee89aafc3f3d3ba1e2bd33531ac7945297e1bcdef2a163f\": container with ID starting with c0239b5d4da420c21ee89aafc3f3d3ba1e2bd33531ac7945297e1bcdef2a163f not found: ID does not exist"
Sep 15 11:57:50.681722 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.681697    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"]
Sep 15 11:57:50.687472 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:50.687452    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2ff8fe-659ffdd486-tk9g8"]
Sep 15 11:57:51.730673 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:51.730630    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15e0263b-66ce-4f05-9799-8c257bcbd8be" path="/var/lib/kubelet/pods/15e0263b-66ce-4f05-9799-8c257bcbd8be/volumes"
Sep 15 11:57:51.731193 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:51.731177    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="703d43bb-0028-4c42-a0dc-880819b52a3d" path="/var/lib/kubelet/pods/703d43bb-0028-4c42-a0dc-880819b52a3d/volumes"
Sep 15 11:57:53.324788 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.324751    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq"]
Sep 15 11:57:53.325173 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.325151    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e0263b-66ce-4f05-9799-8c257bcbd8be" containerName="payload-pre-processing"
Sep 15 11:57:53.325173 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.325163    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e0263b-66ce-4f05-9799-8c257bcbd8be" containerName="payload-pre-processing"
Sep 15 11:57:53.325173 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.325172    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="703d43bb-0028-4c42-a0dc-880819b52a3d" containerName="revoke-keys"
Sep 15 11:57:53.325275 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.325178    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="703d43bb-0028-4c42-a0dc-880819b52a3d" containerName="revoke-keys"
Sep 15 11:57:53.325275 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.325250    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="703d43bb-0028-4c42-a0dc-880819b52a3d" containerName="revoke-keys"
Sep 15 11:57:53.325275 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.325258    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15e0263b-66ce-4f05-9799-8c257bcbd8be" containerName="payload-pre-processing"
Sep 15 11:57:53.329552 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.329536    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq"
Sep 15 11:57:53.331608 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.331579    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:57:53.331608 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.331592    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 11:57:53.335959 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.335932    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq"]
Sep 15 11:57:53.451818 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.451784    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e880abc-ceba-4989-8d1a-467e323403eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq\" (UID: \"9e880abc-ceba-4989-8d1a-467e323403eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq"
Sep 15 11:57:53.552515 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.552481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e880abc-ceba-4989-8d1a-467e323403eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq\" (UID: \"9e880abc-ceba-4989-8d1a-467e323403eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq"
Sep 15 11:57:53.553112 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.553095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e880abc-ceba-4989-8d1a-467e323403eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq\" (UID: \"9e880abc-ceba-4989-8d1a-467e323403eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq"
Sep 15 11:57:53.641020 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.640945    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq"
Sep 15 11:57:53.914133 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:53.914105    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq"]
Sep 15 11:57:53.915282 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:57:53.915250    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e880abc_ceba_4989_8d1a_467e323403eb.slice/crio-b39fe5bdafddce1a324135ed0bed438843c2a0b5993344a2ae8da7fea538f52b WatchSource:0}: Error finding container b39fe5bdafddce1a324135ed0bed438843c2a0b5993344a2ae8da7fea538f52b: Status 404 returned error can't find the container with id b39fe5bdafddce1a324135ed0bed438843c2a0b5993344a2ae8da7fea538f52b
Sep 15 11:57:54.685769 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:54.685736    2580 generic.go:356] "Generic (PLEG): container finished" podID="9e880abc-ceba-4989-8d1a-467e323403eb" containerID="fee6de98ddd33b66861422c7f297cfbcda7e53a7526ec280b5a70af97a31cc8e" exitCode=0
Sep 15 11:57:54.686113 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:54.685822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq" event={"ID":"9e880abc-ceba-4989-8d1a-467e323403eb","Type":"ContainerDied","Data":"fee6de98ddd33b66861422c7f297cfbcda7e53a7526ec280b5a70af97a31cc8e"}
Sep 15 11:57:54.686113 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:54.685930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq" event={"ID":"9e880abc-ceba-4989-8d1a-467e323403eb","Type":"ContainerStarted","Data":"b39fe5bdafddce1a324135ed0bed438843c2a0b5993344a2ae8da7fea538f52b"}
Sep 15 11:57:55.825564 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:55.825538    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq"
Sep 15 11:57:55.976619 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:55.976536    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e880abc-ceba-4989-8d1a-467e323403eb-maas-api-service-ca\") pod \"9e880abc-ceba-4989-8d1a-467e323403eb\" (UID: \"9e880abc-ceba-4989-8d1a-467e323403eb\") "
Sep 15 11:57:55.976885 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:55.976847    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e880abc-ceba-4989-8d1a-467e323403eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e880abc-ceba-4989-8d1a-467e323403eb" (UID: "9e880abc-ceba-4989-8d1a-467e323403eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:57:56.077951 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:56.077907    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e880abc-ceba-4989-8d1a-467e323403eb-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:57:56.699103 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:56.699070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq" event={"ID":"9e880abc-ceba-4989-8d1a-467e323403eb","Type":"ContainerDied","Data":"b39fe5bdafddce1a324135ed0bed438843c2a0b5993344a2ae8da7fea538f52b"}
Sep 15 11:57:56.699312 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:56.699120    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b39fe5bdafddce1a324135ed0bed438843c2a0b5993344a2ae8da7fea538f52b"
Sep 15 11:57:56.699312 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:56.699126    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq"
Sep 15 11:57:58.668155 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:58.668111    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b" podUID="86c40c3e-f126-4d73-8988-b0b92977d4c9" containerName="main" containerID="cri-o://7b31b71bcdf6044b59821a2cb6b4980630a73f37b202a5f0143ccb3a1247e214" gracePeriod=45
Sep 15 11:57:58.921382 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:58.921317    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:59.005141 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.005108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-home\") pod \"86c40c3e-f126-4d73-8988-b0b92977d4c9\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") "
Sep 15 11:57:59.005330 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.005171    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-model-cache\") pod \"86c40c3e-f126-4d73-8988-b0b92977d4c9\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") "
Sep 15 11:57:59.005330 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.005305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-tmp-dir\") pod \"86c40c3e-f126-4d73-8988-b0b92977d4c9\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") "
Sep 15 11:57:59.005464 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.005348    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86c40c3e-f126-4d73-8988-b0b92977d4c9-tls-certs\") pod \"86c40c3e-f126-4d73-8988-b0b92977d4c9\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") "
Sep 15 11:57:59.005464 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.005371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-home" (OuterVolumeSpecName: "home") pod "86c40c3e-f126-4d73-8988-b0b92977d4c9" (UID: "86c40c3e-f126-4d73-8988-b0b92977d4c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:57:59.005464 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.005412    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-dshm\") pod \"86c40c3e-f126-4d73-8988-b0b92977d4c9\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") "
Sep 15 11:57:59.005464 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.005445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlpg4\" (UniqueName: \"kubernetes.io/projected/86c40c3e-f126-4d73-8988-b0b92977d4c9-kube-api-access-nlpg4\") pod \"86c40c3e-f126-4d73-8988-b0b92977d4c9\" (UID: \"86c40c3e-f126-4d73-8988-b0b92977d4c9\") "
Sep 15 11:57:59.005643 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.005471    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "86c40c3e-f126-4d73-8988-b0b92977d4c9" (UID: "86c40c3e-f126-4d73-8988-b0b92977d4c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:57:59.005643 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.005543    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86c40c3e-f126-4d73-8988-b0b92977d4c9" (UID: "86c40c3e-f126-4d73-8988-b0b92977d4c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:57:59.005848 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.005818    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:57:59.005848 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.005848    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:57:59.006027 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.005861    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:57:59.007619 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.007597    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c40c3e-f126-4d73-8988-b0b92977d4c9-kube-api-access-nlpg4" (OuterVolumeSpecName: "kube-api-access-nlpg4") pod "86c40c3e-f126-4d73-8988-b0b92977d4c9" (UID: "86c40c3e-f126-4d73-8988-b0b92977d4c9"). InnerVolumeSpecName "kube-api-access-nlpg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:57:59.008000 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.007977    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c40c3e-f126-4d73-8988-b0b92977d4c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86c40c3e-f126-4d73-8988-b0b92977d4c9" (UID: "86c40c3e-f126-4d73-8988-b0b92977d4c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:57:59.008090 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.008066    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-dshm" (OuterVolumeSpecName: "dshm") pod "86c40c3e-f126-4d73-8988-b0b92977d4c9" (UID: "86c40c3e-f126-4d73-8988-b0b92977d4c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:57:59.107248 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.107214    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlpg4\" (UniqueName: \"kubernetes.io/projected/86c40c3e-f126-4d73-8988-b0b92977d4c9-kube-api-access-nlpg4\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:57:59.107248 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.107244    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86c40c3e-f126-4d73-8988-b0b92977d4c9-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:57:59.107248 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.107253    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86c40c3e-f126-4d73-8988-b0b92977d4c9-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:57:59.725022 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.724989    2580 generic.go:356] "Generic (PLEG): container finished" podID="86c40c3e-f126-4d73-8988-b0b92977d4c9" containerID="7b31b71bcdf6044b59821a2cb6b4980630a73f37b202a5f0143ccb3a1247e214" exitCode=0
Sep 15 11:57:59.725407 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.725107    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"
Sep 15 11:57:59.730634 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.730609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b" event={"ID":"86c40c3e-f126-4d73-8988-b0b92977d4c9","Type":"ContainerDied","Data":"7b31b71bcdf6044b59821a2cb6b4980630a73f37b202a5f0143ccb3a1247e214"}
Sep 15 11:57:59.730780 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.730638    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b" event={"ID":"86c40c3e-f126-4d73-8988-b0b92977d4c9","Type":"ContainerDied","Data":"8a8a591f506c571959533f871d987bdfb0e68597e3c3b2bc3426ba124164c613"}
Sep 15 11:57:59.730780 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.730657    2580 scope.go:117] "RemoveContainer" containerID="7b31b71bcdf6044b59821a2cb6b4980630a73f37b202a5f0143ccb3a1247e214"
Sep 15 11:57:59.740364 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.740346    2580 scope.go:117] "RemoveContainer" containerID="7b31b71bcdf6044b59821a2cb6b4980630a73f37b202a5f0143ccb3a1247e214"
Sep 15 11:57:59.740618 ip-10-0-143-211 kubenswrapper[2580]: E0915 11:57:59.740599    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b31b71bcdf6044b59821a2cb6b4980630a73f37b202a5f0143ccb3a1247e214\": container with ID starting with 7b31b71bcdf6044b59821a2cb6b4980630a73f37b202a5f0143ccb3a1247e214 not found: ID does not exist" containerID="7b31b71bcdf6044b59821a2cb6b4980630a73f37b202a5f0143ccb3a1247e214"
Sep 15 11:57:59.740681 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.740624    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b31b71bcdf6044b59821a2cb6b4980630a73f37b202a5f0143ccb3a1247e214"} err="failed to get container status \"7b31b71bcdf6044b59821a2cb6b4980630a73f37b202a5f0143ccb3a1247e214\": rpc error: code = NotFound desc = could not find container \"7b31b71bcdf6044b59821a2cb6b4980630a73f37b202a5f0143ccb3a1247e214\": container with ID starting with 7b31b71bcdf6044b59821a2cb6b4980630a73f37b202a5f0143ccb3a1247e214 not found: ID does not exist"
Sep 15 11:57:59.749414 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.749389    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"]
Sep 15 11:57:59.754892 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:57:59.754858    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-2ff8fe/explicit-model-2ff8fe-kserve-648d68fc59-96b5b"]
Sep 15 11:58:01.731270 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:01.731239    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86c40c3e-f126-4d73-8988-b0b92977d4c9" path="/var/lib/kubelet/pods/86c40c3e-f126-4d73-8988-b0b92977d4c9/volumes"
Sep 15 11:58:04.450536 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:04.450503    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq"]
Sep 15 11:58:04.455973 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:04.455946    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cdf667d8-48bdq"]
Sep 15 11:58:05.731468 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:05.731428    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e880abc-ceba-4989-8d1a-467e323403eb" path="/var/lib/kubelet/pods/9e880abc-ceba-4989-8d1a-467e323403eb/volumes"
Sep 15 11:58:34.317494 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.317458    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"]
Sep 15 11:58:34.317913 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.317838    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c40c3e-f126-4d73-8988-b0b92977d4c9" containerName="main"
Sep 15 11:58:34.317913 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.317852    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c40c3e-f126-4d73-8988-b0b92977d4c9" containerName="main"
Sep 15 11:58:34.317913 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.317867    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e880abc-ceba-4989-8d1a-467e323403eb" containerName="revoke-keys"
Sep 15 11:58:34.317913 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.317888    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e880abc-ceba-4989-8d1a-467e323403eb" containerName="revoke-keys"
Sep 15 11:58:34.318052 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.317971    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e880abc-ceba-4989-8d1a-467e323403eb" containerName="revoke-keys"
Sep 15 11:58:34.318052 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.317980    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c40c3e-f126-4d73-8988-b0b92977d4c9" containerName="main"
Sep 15 11:58:34.322439 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.322419    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"
Sep 15 11:58:34.325257 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.325236    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ec8230-dockercfg-p5tpb\""
Sep 15 11:58:34.325384 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.325318    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ec8230\""
Sep 15 11:58:34.332415 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.332386    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"]
Sep 15 11:58:34.448827 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.448787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57xtm\" (UniqueName: \"kubernetes.io/projected/e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c-kube-api-access-57xtm\") pod \"payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr\" (UID: \"e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"
Sep 15 11:58:34.449064 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.448992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr\" (UID: \"e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"
Sep 15 11:58:34.550457 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.550419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr\" (UID: \"e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"
Sep 15 11:58:34.550649 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.550486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57xtm\" (UniqueName: \"kubernetes.io/projected/e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c-kube-api-access-57xtm\") pod \"payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr\" (UID: \"e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"
Sep 15 11:58:34.551182 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.551160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr\" (UID: \"e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"
Sep 15 11:58:34.559675 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.559648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57xtm\" (UniqueName: \"kubernetes.io/projected/e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c-kube-api-access-57xtm\") pod \"payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr\" (UID: \"e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"
Sep 15 11:58:34.633810 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.633746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"
Sep 15 11:58:34.912422 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.912393    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"]
Sep 15 11:58:34.913387 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:58:34.913363    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10f1f44_4abe_4b7e_949d_f6e1b4cfdc9c.slice/crio-b3c334597db7f6baec1a471436722ee9dfe6182c42452772351c1e7efd90c8c1 WatchSource:0}: Error finding container b3c334597db7f6baec1a471436722ee9dfe6182c42452772351c1e7efd90c8c1: Status 404 returned error can't find the container with id b3c334597db7f6baec1a471436722ee9dfe6182c42452772351c1e7efd90c8c1
Sep 15 11:58:34.930553 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:34.930524    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr" event={"ID":"e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c","Type":"ContainerStarted","Data":"b3c334597db7f6baec1a471436722ee9dfe6182c42452772351c1e7efd90c8c1"}
Sep 15 11:58:35.937698 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:35.937660    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr" event={"ID":"e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c","Type":"ContainerStarted","Data":"f9448f0d3c1820d7f68d20d6b8d8e57f0892cc935a7877023e2f9acec51cc616"}
Sep 15 11:58:35.938194 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:35.937831    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"
Sep 15 11:58:35.957487 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:35.957440    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr" podStartSLOduration=1.9574223499999999 podStartE2EDuration="1.95742235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:58:35.954403027 +0000 UTC m=+1204.734974414" watchObservedRunningTime="2026-09-15 11:58:35.95742235 +0000 UTC m=+1204.737993737"
Sep 15 11:58:46.945030 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:58:46.944998    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"
Sep 15 11:59:09.459384 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:09.459342    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"]
Sep 15 11:59:09.462923 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:09.462905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"
Sep 15 11:59:09.465339 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:09.465317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6dca39-dockercfg-4bl8b\""
Sep 15 11:59:09.465429 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:09.465320    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6dca39\""
Sep 15 11:59:09.475642 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:09.475616    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"]
Sep 15 11:59:09.605848 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:09.605815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz9vf\" (UniqueName: \"kubernetes.io/projected/c6978c6b-5b97-46a5-b3b6-8b7d81e141f1-kube-api-access-fz9vf\") pod \"payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h\" (UID: \"c6978c6b-5b97-46a5-b3b6-8b7d81e141f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"
Sep 15 11:59:09.606031 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:09.605885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6978c6b-5b97-46a5-b3b6-8b7d81e141f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h\" (UID: \"c6978c6b-5b97-46a5-b3b6-8b7d81e141f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"
Sep 15 11:59:09.706854 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:09.706814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz9vf\" (UniqueName: \"kubernetes.io/projected/c6978c6b-5b97-46a5-b3b6-8b7d81e141f1-kube-api-access-fz9vf\") pod \"payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h\" (UID: \"c6978c6b-5b97-46a5-b3b6-8b7d81e141f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"
Sep 15 11:59:09.707073 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:09.706976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6978c6b-5b97-46a5-b3b6-8b7d81e141f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h\" (UID: \"c6978c6b-5b97-46a5-b3b6-8b7d81e141f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"
Sep 15 11:59:09.707712 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:09.707687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6978c6b-5b97-46a5-b3b6-8b7d81e141f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h\" (UID: \"c6978c6b-5b97-46a5-b3b6-8b7d81e141f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"
Sep 15 11:59:09.716137 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:09.716063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz9vf\" (UniqueName: \"kubernetes.io/projected/c6978c6b-5b97-46a5-b3b6-8b7d81e141f1-kube-api-access-fz9vf\") pod \"payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h\" (UID: \"c6978c6b-5b97-46a5-b3b6-8b7d81e141f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"
Sep 15 11:59:09.773319 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:09.773287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"
Sep 15 11:59:10.047065 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:10.047039    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"]
Sep 15 11:59:10.049351 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:59:10.049320    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6978c6b_5b97_46a5_b3b6_8b7d81e141f1.slice/crio-ad77af8c656385e0cc68e4ad136ef130ce0b87d9b2230b29c54b2f370addf086 WatchSource:0}: Error finding container ad77af8c656385e0cc68e4ad136ef130ce0b87d9b2230b29c54b2f370addf086: Status 404 returned error can't find the container with id ad77af8c656385e0cc68e4ad136ef130ce0b87d9b2230b29c54b2f370addf086
Sep 15 11:59:10.137159 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:10.137122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h" event={"ID":"c6978c6b-5b97-46a5-b3b6-8b7d81e141f1","Type":"ContainerStarted","Data":"2078637080ab95bead9eda00437931a78b36501495931abdc22f1aa8e69bb240"}
Sep 15 11:59:10.137292 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:10.137182    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h" event={"ID":"c6978c6b-5b97-46a5-b3b6-8b7d81e141f1","Type":"ContainerStarted","Data":"ad77af8c656385e0cc68e4ad136ef130ce0b87d9b2230b29c54b2f370addf086"}
Sep 15 11:59:10.137292 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:10.137267    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"
Sep 15 11:59:10.154407 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:10.154363    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h" podStartSLOduration=1.1543474 podStartE2EDuration="1.1543474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:59:10.153078427 +0000 UTC m=+1238.933649822" watchObservedRunningTime="2026-09-15 11:59:10.1543474 +0000 UTC m=+1238.934918788"
Sep 15 11:59:21.144887 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:21.144841    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"
Sep 15 11:59:31.815823 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:31.815751    2580 scope.go:117] "RemoveContainer" containerID="c6b2f9ce59641f4625f11dc99c505f1dc14ac903c36ac76bd1ced7ccb8f5b66f"
Sep 15 11:59:52.048930 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.048891    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"]
Sep 15 11:59:52.053814 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.053793    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"
Sep 15 11:59:52.055999 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.055980    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cc097cc5\""
Sep 15 11:59:52.056095 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.055998    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cc097cc5-dockercfg-s9fbp\""
Sep 15 11:59:52.064037 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.064011    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"]
Sep 15 11:59:52.075177 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.075148    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"]
Sep 15 11:59:52.080161 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.080132    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"
Sep 15 11:59:52.089378 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.089354    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"]
Sep 15 11:59:52.163417 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.163383    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf"]
Sep 15 11:59:52.166955 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.166932    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf"
Sep 15 11:59:52.169256 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.169234    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:59:52.169392 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.169236    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 11:59:52.176696 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.176668    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf"]
Sep 15 11:59:52.227362 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.227325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d868b5-01cd-410a-87da-908ec2aacc64-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96\" (UID: \"14d868b5-01cd-410a-87da-908ec2aacc64\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"
Sep 15 11:59:52.227523 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.227431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95lnh\" (UniqueName: \"kubernetes.io/projected/14d868b5-01cd-410a-87da-908ec2aacc64-kube-api-access-95lnh\") pod \"payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96\" (UID: \"14d868b5-01cd-410a-87da-908ec2aacc64\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"
Sep 15 11:59:52.227523 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.227493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b26bff-dc0e-4f72-9951-3d2b638aaedd-plugins-config-volume\") pod \"payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs\" (UID: \"d1b26bff-dc0e-4f72-9951-3d2b638aaedd\") " pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"
Sep 15 11:59:52.227599 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.227528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blrk4\" (UniqueName: \"kubernetes.io/projected/d1b26bff-dc0e-4f72-9951-3d2b638aaedd-kube-api-access-blrk4\") pod \"payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs\" (UID: \"d1b26bff-dc0e-4f72-9951-3d2b638aaedd\") " pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"
Sep 15 11:59:52.328732 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.328650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d868b5-01cd-410a-87da-908ec2aacc64-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96\" (UID: \"14d868b5-01cd-410a-87da-908ec2aacc64\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"
Sep 15 11:59:52.328732 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.328697    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1965e277-a7ba-4044-b8fe-b969650c849a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf\" (UID: \"1965e277-a7ba-4044-b8fe-b969650c849a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf"
Sep 15 11:59:52.328975 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.328735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95lnh\" (UniqueName: \"kubernetes.io/projected/14d868b5-01cd-410a-87da-908ec2aacc64-kube-api-access-95lnh\") pod \"payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96\" (UID: \"14d868b5-01cd-410a-87da-908ec2aacc64\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"
Sep 15 11:59:52.328975 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.328767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b26bff-dc0e-4f72-9951-3d2b638aaedd-plugins-config-volume\") pod \"payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs\" (UID: \"d1b26bff-dc0e-4f72-9951-3d2b638aaedd\") " pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"
Sep 15 11:59:52.328975 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.328796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blrk4\" (UniqueName: \"kubernetes.io/projected/d1b26bff-dc0e-4f72-9951-3d2b638aaedd-kube-api-access-blrk4\") pod \"payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs\" (UID: \"d1b26bff-dc0e-4f72-9951-3d2b638aaedd\") " pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"
Sep 15 11:59:52.329466 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.329446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d868b5-01cd-410a-87da-908ec2aacc64-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96\" (UID: \"14d868b5-01cd-410a-87da-908ec2aacc64\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"
Sep 15 11:59:52.329466 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.329457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b26bff-dc0e-4f72-9951-3d2b638aaedd-plugins-config-volume\") pod \"payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs\" (UID: \"d1b26bff-dc0e-4f72-9951-3d2b638aaedd\") " pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"
Sep 15 11:59:52.338021 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.338001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blrk4\" (UniqueName: \"kubernetes.io/projected/d1b26bff-dc0e-4f72-9951-3d2b638aaedd-kube-api-access-blrk4\") pod \"payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs\" (UID: \"d1b26bff-dc0e-4f72-9951-3d2b638aaedd\") " pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"
Sep 15 11:59:52.338123 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.338055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95lnh\" (UniqueName: \"kubernetes.io/projected/14d868b5-01cd-410a-87da-908ec2aacc64-kube-api-access-95lnh\") pod \"payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96\" (UID: \"14d868b5-01cd-410a-87da-908ec2aacc64\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"
Sep 15 11:59:52.365945 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.365917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"
Sep 15 11:59:52.393860 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.393830    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"
Sep 15 11:59:52.430158 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.430125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1965e277-a7ba-4044-b8fe-b969650c849a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf\" (UID: \"1965e277-a7ba-4044-b8fe-b969650c849a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf"
Sep 15 11:59:52.431002 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.430971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1965e277-a7ba-4044-b8fe-b969650c849a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf\" (UID: \"1965e277-a7ba-4044-b8fe-b969650c849a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf"
Sep 15 11:59:52.481228 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.481204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf"
Sep 15 11:59:52.658794 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.658714    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"]
Sep 15 11:59:52.661896 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:59:52.661843    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d868b5_01cd_410a_87da_908ec2aacc64.slice/crio-ad92585fb572f88f4e3e6f8724e16454b7d5920549851ef9a8b6896e1ad6e1ec WatchSource:0}: Error finding container ad92585fb572f88f4e3e6f8724e16454b7d5920549851ef9a8b6896e1ad6e1ec: Status 404 returned error can't find the container with id ad92585fb572f88f4e3e6f8724e16454b7d5920549851ef9a8b6896e1ad6e1ec
Sep 15 11:59:52.686931 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.686804    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"]
Sep 15 11:59:52.695992 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:59:52.695970    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b26bff_dc0e_4f72_9951_3d2b638aaedd.slice/crio-b031515db5b04b1d46536025f4bff05ce0137be60fc5195494be1c506046f7b0 WatchSource:0}: Error finding container b031515db5b04b1d46536025f4bff05ce0137be60fc5195494be1c506046f7b0: Status 404 returned error can't find the container with id b031515db5b04b1d46536025f4bff05ce0137be60fc5195494be1c506046f7b0
Sep 15 11:59:52.763362 ip-10-0-143-211 kubenswrapper[2580]: W0915 11:59:52.763333    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1965e277_a7ba_4044_b8fe_b969650c849a.slice/crio-66e18e18e4801cc9105c1331e629017d685fc922a2ec480c67813a29ee1a3626 WatchSource:0}: Error finding container 66e18e18e4801cc9105c1331e629017d685fc922a2ec480c67813a29ee1a3626: Status 404 returned error can't find the container with id 66e18e18e4801cc9105c1331e629017d685fc922a2ec480c67813a29ee1a3626
Sep 15 11:59:52.763644 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:52.763616    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf"]
Sep 15 11:59:53.401823 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:53.401724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs" event={"ID":"d1b26bff-dc0e-4f72-9951-3d2b638aaedd","Type":"ContainerStarted","Data":"c0c2346f051b9ceec805a8ba4103237a9da5cca312a5a3f9bf1e24f43d3e8402"}
Sep 15 11:59:53.401823 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:53.401796    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs" event={"ID":"d1b26bff-dc0e-4f72-9951-3d2b638aaedd","Type":"ContainerStarted","Data":"b031515db5b04b1d46536025f4bff05ce0137be60fc5195494be1c506046f7b0"}
Sep 15 11:59:53.402275 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:53.401920    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"
Sep 15 11:59:53.403454 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:53.403427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96" event={"ID":"14d868b5-01cd-410a-87da-908ec2aacc64","Type":"ContainerStarted","Data":"261353cf556d84e692cd11646e3b586900e59107527d697263f9367936b7935e"}
Sep 15 11:59:53.403547 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:53.403481    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96" event={"ID":"14d868b5-01cd-410a-87da-908ec2aacc64","Type":"ContainerStarted","Data":"ad92585fb572f88f4e3e6f8724e16454b7d5920549851ef9a8b6896e1ad6e1ec"}
Sep 15 11:59:53.403547 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:53.403510    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"
Sep 15 11:59:53.405056 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:53.405033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf" event={"ID":"1965e277-a7ba-4044-b8fe-b969650c849a","Type":"ContainerStarted","Data":"47d612e48b3c7cbf404f78fd21d8e65cdd660fc75e9c09a18cae2eafc544f43b"}
Sep 15 11:59:53.405143 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:53.405080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf" event={"ID":"1965e277-a7ba-4044-b8fe-b969650c849a","Type":"ContainerStarted","Data":"66e18e18e4801cc9105c1331e629017d685fc922a2ec480c67813a29ee1a3626"}
Sep 15 11:59:53.419467 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:53.419425    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs" podStartSLOduration=1.419413573 podStartE2EDuration="1.419413573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:59:53.418218346 +0000 UTC m=+1282.198789733" watchObservedRunningTime="2026-09-15 11:59:53.419413573 +0000 UTC m=+1282.199984961"
Sep 15 11:59:53.436930 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:53.436862    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96" podStartSLOduration=1.436849229 podStartE2EDuration="1.436849229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:59:53.436507929 +0000 UTC m=+1282.217079315" watchObservedRunningTime="2026-09-15 11:59:53.436849229 +0000 UTC m=+1282.217420639"
Sep 15 11:59:53.451568 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:53.451526    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf" podStartSLOduration=1.451508183 podStartE2EDuration="1.451508183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:59:53.450690074 +0000 UTC m=+1282.231261446" watchObservedRunningTime="2026-09-15 11:59:53.451508183 +0000 UTC m=+1282.232079570"
Sep 15 11:59:54.412108 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:54.412070    2580 generic.go:356] "Generic (PLEG): container finished" podID="1965e277-a7ba-4044-b8fe-b969650c849a" containerID="47d612e48b3c7cbf404f78fd21d8e65cdd660fc75e9c09a18cae2eafc544f43b" exitCode=0
Sep 15 11:59:54.412510 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:54.412156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf" event={"ID":"1965e277-a7ba-4044-b8fe-b969650c849a","Type":"ContainerDied","Data":"47d612e48b3c7cbf404f78fd21d8e65cdd660fc75e9c09a18cae2eafc544f43b"}
Sep 15 11:59:55.554631 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:55.554609    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf"
Sep 15 11:59:55.663032 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:55.663002    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1965e277-a7ba-4044-b8fe-b969650c849a-maas-api-service-ca\") pod \"1965e277-a7ba-4044-b8fe-b969650c849a\" (UID: \"1965e277-a7ba-4044-b8fe-b969650c849a\") "
Sep 15 11:59:55.663353 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:55.663332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1965e277-a7ba-4044-b8fe-b969650c849a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1965e277-a7ba-4044-b8fe-b969650c849a" (UID: "1965e277-a7ba-4044-b8fe-b969650c849a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:59:55.763994 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:55.763960    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1965e277-a7ba-4044-b8fe-b969650c849a-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 11:59:56.426938 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:56.426906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf" event={"ID":"1965e277-a7ba-4044-b8fe-b969650c849a","Type":"ContainerDied","Data":"66e18e18e4801cc9105c1331e629017d685fc922a2ec480c67813a29ee1a3626"}
Sep 15 11:59:56.427101 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:56.426919    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf"
Sep 15 11:59:56.427101 ip-10-0-143-211 kubenswrapper[2580]: I0915 11:59:56.426964    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e18e18e4801cc9105c1331e629017d685fc922a2ec480c67813a29ee1a3626"
Sep 15 12:00:00.144559 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.144520    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds"]
Sep 15 12:00:00.145046 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.145030    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1965e277-a7ba-4044-b8fe-b969650c849a" containerName="revoke-keys"
Sep 15 12:00:00.145113 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.145048    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1965e277-a7ba-4044-b8fe-b969650c849a" containerName="revoke-keys"
Sep 15 12:00:00.145159 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.145146    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1965e277-a7ba-4044-b8fe-b969650c849a" containerName="revoke-keys"
Sep 15 12:00:00.148441 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.148422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds"
Sep 15 12:00:00.151274 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.151255    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:00:00.155399 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.155374    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac76e5-29824560-fjvmh"]
Sep 15 12:00:00.159691 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.159670    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac76e5-29824560-fjvmh"
Sep 15 12:00:00.161360 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.161342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds"
Sep 15 12:00:00.162792 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.162764    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-29d630-29824560-wpv5n"]
Sep 15 12:00:00.168774 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.168753    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds"]
Sep 15 12:00:00.169100 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.169081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-29d630-29824560-wpv5n"
Sep 15 12:00:00.176669 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.176640    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac76e5-29824560-fjvmh"]
Sep 15 12:00:00.180142 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.180114    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ec8230-29824560-rnwkq"]
Sep 15 12:00:00.184794 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.184727    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ec8230-29824560-rnwkq"
Sep 15 12:00:00.185237 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.185216    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-29d630-29824560-wpv5n"]
Sep 15 12:00:00.194056 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.194019    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ec8230-29824560-rnwkq"]
Sep 15 12:00:00.198568 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.198498    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6dca39-29824560-wkqvn"]
Sep 15 12:00:00.204751 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.204728    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6dca39-29824560-wkqvn"
Sep 15 12:00:00.207459 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.207419    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac76e5-29824560-fjvmh"
Sep 15 12:00:00.207459 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.207452    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6dca39-29824560-wkqvn"]
Sep 15 12:00:00.218576 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.218552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-29d630-29824560-wpv5n"
Sep 15 12:00:00.272681 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.272475    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ec8230-29824560-rnwkq"
Sep 15 12:00:00.283914 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.283889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6dca39-29824560-wkqvn"
Sep 15 12:00:00.464240 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.464209    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds"]
Sep 15 12:00:00.464443 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:00:00.464419    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9419a91e_0dc3_45b2_b3d6_0a9f0c7846a0.slice/crio-f3322150b1324bb6a42b9897d4cba13fc91267195d0a100ac12b21fdae50bc0e WatchSource:0}: Error finding container f3322150b1324bb6a42b9897d4cba13fc91267195d0a100ac12b21fdae50bc0e: Status 404 returned error can't find the container with id f3322150b1324bb6a42b9897d4cba13fc91267195d0a100ac12b21fdae50bc0e
Sep 15 12:00:00.527654 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.527624    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac76e5-29824560-fjvmh"]
Sep 15 12:00:00.527654 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:00:00.527640    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e97acc_8531_4a9a_b068_7425fc3a4434.slice/crio-858e59edbeae8707acbfce98f5a498f9d913198fbf8009e0bd663f359babc1af WatchSource:0}: Error finding container 858e59edbeae8707acbfce98f5a498f9d913198fbf8009e0bd663f359babc1af: Status 404 returned error can't find the container with id 858e59edbeae8707acbfce98f5a498f9d913198fbf8009e0bd663f359babc1af
Sep 15 12:00:00.533451 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.533425    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-29d630-29824560-wpv5n"]
Sep 15 12:00:00.534044 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:00:00.534025    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67547c19_52b2_4be3_8c21_7aa2253b647b.slice/crio-80b4a1ee7a1ab142c91c0f46fee8876c64d213697ad8f9109ef097b433ecfd1d WatchSource:0}: Error finding container 80b4a1ee7a1ab142c91c0f46fee8876c64d213697ad8f9109ef097b433ecfd1d: Status 404 returned error can't find the container with id 80b4a1ee7a1ab142c91c0f46fee8876c64d213697ad8f9109ef097b433ecfd1d
Sep 15 12:00:00.594155 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.594121    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6dca39-29824560-wkqvn"]
Sep 15 12:00:00.596055 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:00:00.596028    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54ccce0_837b_4b9b_867d_5c9d45e33719.slice/crio-a0e3cbd407974f18080139c4a59d7f9a94c8ad9fa207e504359e1d28fe6332fa WatchSource:0}: Error finding container a0e3cbd407974f18080139c4a59d7f9a94c8ad9fa207e504359e1d28fe6332fa: Status 404 returned error can't find the container with id a0e3cbd407974f18080139c4a59d7f9a94c8ad9fa207e504359e1d28fe6332fa
Sep 15 12:00:00.597026 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:00.596999    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ec8230-29824560-rnwkq"]
Sep 15 12:00:00.598062 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:00:00.598040    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52dc0ba_2fc3_44ba_b964_071011b0e912.slice/crio-d0e0b03c807769347e382fe1525bdbd37ee2916967e9fb69189099672c678c63 WatchSource:0}: Error finding container d0e0b03c807769347e382fe1525bdbd37ee2916967e9fb69189099672c678c63: Status 404 returned error can't find the container with id d0e0b03c807769347e382fe1525bdbd37ee2916967e9fb69189099672c678c63
Sep 15 12:00:01.459592 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.459550    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds" event={"ID":"9419a91e-0dc3-45b2-b3d6-0a9f0c7846a0","Type":"ContainerStarted","Data":"a03080c439155165b774d3fcacd1d2f492220f451bf7a2a9f178b7f78576f382"}
Sep 15 12:00:01.460064 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.459617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds" event={"ID":"9419a91e-0dc3-45b2-b3d6-0a9f0c7846a0","Type":"ContainerStarted","Data":"f3322150b1324bb6a42b9897d4cba13fc91267195d0a100ac12b21fdae50bc0e"}
Sep 15 12:00:01.461315 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.461289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-29d630-29824560-wpv5n" event={"ID":"67547c19-52b2-4be3-8c21-7aa2253b647b","Type":"ContainerStarted","Data":"8e44e5eccedafc16bbb433cec347a77b0407471aa232d0745a5be8d1c3d1f43c"}
Sep 15 12:00:01.461446 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.461332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-29d630-29824560-wpv5n" event={"ID":"67547c19-52b2-4be3-8c21-7aa2253b647b","Type":"ContainerStarted","Data":"80b4a1ee7a1ab142c91c0f46fee8876c64d213697ad8f9109ef097b433ecfd1d"}
Sep 15 12:00:01.463019 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.462994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac76e5-29824560-fjvmh" event={"ID":"a6e97acc-8531-4a9a-b068-7425fc3a4434","Type":"ContainerStarted","Data":"0cd7de93fade0e5942d7d4e65c235da46856e239861dd7fef96fc7bbd5125e29"}
Sep 15 12:00:01.463114 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.463040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac76e5-29824560-fjvmh" event={"ID":"a6e97acc-8531-4a9a-b068-7425fc3a4434","Type":"ContainerStarted","Data":"858e59edbeae8707acbfce98f5a498f9d913198fbf8009e0bd663f359babc1af"}
Sep 15 12:00:01.464728 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.464703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6dca39-29824560-wkqvn" event={"ID":"c54ccce0-837b-4b9b-867d-5c9d45e33719","Type":"ContainerStarted","Data":"e43b7f846dd9f4506ee5ca8798b6bc5d9c3671b4e4dcc779c52d405b04f2cbd9"}
Sep 15 12:00:01.464850 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.464752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6dca39-29824560-wkqvn" event={"ID":"c54ccce0-837b-4b9b-867d-5c9d45e33719","Type":"ContainerStarted","Data":"a0e3cbd407974f18080139c4a59d7f9a94c8ad9fa207e504359e1d28fe6332fa"}
Sep 15 12:00:01.466386 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.466359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ec8230-29824560-rnwkq" event={"ID":"d52dc0ba-2fc3-44ba-b964-071011b0e912","Type":"ContainerStarted","Data":"c42149d494e9e48dc57341ffff2cb317dfc3ba53a6683917449aee06139f135b"}
Sep 15 12:00:01.466478 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.466403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ec8230-29824560-rnwkq" event={"ID":"d52dc0ba-2fc3-44ba-b964-071011b0e912","Type":"ContainerStarted","Data":"d0e0b03c807769347e382fe1525bdbd37ee2916967e9fb69189099672c678c63"}
Sep 15 12:00:01.491226 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.491187    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds" podStartSLOduration=1.491173106 podStartE2EDuration="1.491173106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:01.489837067 +0000 UTC m=+1290.270408455" watchObservedRunningTime="2026-09-15 12:00:01.491173106 +0000 UTC m=+1290.271744514"
Sep 15 12:00:01.514420 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.514371    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ec8230-29824560-rnwkq" podStartSLOduration=1.514355619 podStartE2EDuration="1.514355619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:01.511864276 +0000 UTC m=+1290.292435672" watchObservedRunningTime="2026-09-15 12:00:01.514355619 +0000 UTC m=+1290.294927007"
Sep 15 12:00:01.535195 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.535151    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac76e5-29824560-fjvmh" podStartSLOduration=1.535137018 podStartE2EDuration="1.535137018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:01.532326613 +0000 UTC m=+1290.312898000" watchObservedRunningTime="2026-09-15 12:00:01.535137018 +0000 UTC m=+1290.315708408"
Sep 15 12:00:01.548009 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.547950    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-29d630-29824560-wpv5n" podStartSLOduration=1.547931233 podStartE2EDuration="1.547931233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:01.547588333 +0000 UTC m=+1290.328159740" watchObservedRunningTime="2026-09-15 12:00:01.547931233 +0000 UTC m=+1290.328502620"
Sep 15 12:00:01.562624 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:01.562579    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6dca39-29824560-wkqvn" podStartSLOduration=1.562560806 podStartE2EDuration="1.562560806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:01.561977951 +0000 UTC m=+1290.342549338" watchObservedRunningTime="2026-09-15 12:00:01.562560806 +0000 UTC m=+1290.343132195"
Sep 15 12:00:02.475140 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:02.475104    2580 generic.go:356] "Generic (PLEG): container finished" podID="a6e97acc-8531-4a9a-b068-7425fc3a4434" containerID="0cd7de93fade0e5942d7d4e65c235da46856e239861dd7fef96fc7bbd5125e29" exitCode=0
Sep 15 12:00:02.475595 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:02.475245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac76e5-29824560-fjvmh" event={"ID":"a6e97acc-8531-4a9a-b068-7425fc3a4434","Type":"ContainerDied","Data":"0cd7de93fade0e5942d7d4e65c235da46856e239861dd7fef96fc7bbd5125e29"}
Sep 15 12:00:02.477402 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:02.477375    2580 generic.go:356] "Generic (PLEG): container finished" podID="c54ccce0-837b-4b9b-867d-5c9d45e33719" containerID="e43b7f846dd9f4506ee5ca8798b6bc5d9c3671b4e4dcc779c52d405b04f2cbd9" exitCode=0
Sep 15 12:00:02.477542 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:02.477462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6dca39-29824560-wkqvn" event={"ID":"c54ccce0-837b-4b9b-867d-5c9d45e33719","Type":"ContainerDied","Data":"e43b7f846dd9f4506ee5ca8798b6bc5d9c3671b4e4dcc779c52d405b04f2cbd9"}
Sep 15 12:00:02.479071 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:02.479049    2580 generic.go:356] "Generic (PLEG): container finished" podID="d52dc0ba-2fc3-44ba-b964-071011b0e912" containerID="c42149d494e9e48dc57341ffff2cb317dfc3ba53a6683917449aee06139f135b" exitCode=0
Sep 15 12:00:02.479165 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:02.479135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ec8230-29824560-rnwkq" event={"ID":"d52dc0ba-2fc3-44ba-b964-071011b0e912","Type":"ContainerDied","Data":"c42149d494e9e48dc57341ffff2cb317dfc3ba53a6683917449aee06139f135b"}
Sep 15 12:00:02.480687 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:02.480667    2580 generic.go:356] "Generic (PLEG): container finished" podID="9419a91e-0dc3-45b2-b3d6-0a9f0c7846a0" containerID="a03080c439155165b774d3fcacd1d2f492220f451bf7a2a9f178b7f78576f382" exitCode=0
Sep 15 12:00:02.480774 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:02.480691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds" event={"ID":"9419a91e-0dc3-45b2-b3d6-0a9f0c7846a0","Type":"ContainerDied","Data":"a03080c439155165b774d3fcacd1d2f492220f451bf7a2a9f178b7f78576f382"}
Sep 15 12:00:02.482417 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:02.482396    2580 generic.go:356] "Generic (PLEG): container finished" podID="67547c19-52b2-4be3-8c21-7aa2253b647b" containerID="8e44e5eccedafc16bbb433cec347a77b0407471aa232d0745a5be8d1c3d1f43c" exitCode=0
Sep 15 12:00:02.482500 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:02.482471    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-29d630-29824560-wpv5n" event={"ID":"67547c19-52b2-4be3-8c21-7aa2253b647b","Type":"ContainerDied","Data":"8e44e5eccedafc16bbb433cec347a77b0407471aa232d0745a5be8d1c3d1f43c"}
Sep 15 12:00:03.178136 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.178096    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds"]
Sep 15 12:00:03.203395 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.203363    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"]
Sep 15 12:00:03.203624 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.203598    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q" podUID="7871a402-06d0-4678-bd1e-1cdda46dd153" containerName="payload-processing" containerID="cri-o://6f9e413166760100ccfc55302c2f5fc0c18b065be1876d0a22c08e0163fb8a9a" gracePeriod=30
Sep 15 12:00:03.212403 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.212372    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"]
Sep 15 12:00:03.212780 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.212576    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq" podUID="e4b4b25f-8bda-4307-a7a5-b6d03a846ad7" containerName="payload-pre-processing" containerID="cri-o://82f03b86614251775906a220a0c687cbf15ec7a2d5e50a899c551c27080447d0" gracePeriod=30
Sep 15 12:00:03.400780 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.400743    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf"]
Sep 15 12:00:03.408985 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.408952    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97d02d-vplhf"]
Sep 15 12:00:03.492549 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.492520    2580 generic.go:356] "Generic (PLEG): container finished" podID="7871a402-06d0-4678-bd1e-1cdda46dd153" containerID="6f9e413166760100ccfc55302c2f5fc0c18b065be1876d0a22c08e0163fb8a9a" exitCode=0
Sep 15 12:00:03.492865 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.492597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q" event={"ID":"7871a402-06d0-4678-bd1e-1cdda46dd153","Type":"ContainerDied","Data":"6f9e413166760100ccfc55302c2f5fc0c18b065be1876d0a22c08e0163fb8a9a"}
Sep 15 12:00:03.494220 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.494197    2580 generic.go:356] "Generic (PLEG): container finished" podID="e4b4b25f-8bda-4307-a7a5-b6d03a846ad7" containerID="82f03b86614251775906a220a0c687cbf15ec7a2d5e50a899c551c27080447d0" exitCode=0
Sep 15 12:00:03.494355 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.494332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq" event={"ID":"e4b4b25f-8bda-4307-a7a5-b6d03a846ad7","Type":"ContainerDied","Data":"82f03b86614251775906a220a0c687cbf15ec7a2d5e50a899c551c27080447d0"}
Sep 15 12:00:03.529926 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.529895    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"
Sep 15 12:00:03.536850 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.536813    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"
Sep 15 12:00:03.644927 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.644895    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4b25f-8bda-4307-a7a5-b6d03a846ad7-plugins-config-volume\") pod \"e4b4b25f-8bda-4307-a7a5-b6d03a846ad7\" (UID: \"e4b4b25f-8bda-4307-a7a5-b6d03a846ad7\") "
Sep 15 12:00:03.645111 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.644946    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8598c\" (UniqueName: \"kubernetes.io/projected/e4b4b25f-8bda-4307-a7a5-b6d03a846ad7-kube-api-access-8598c\") pod \"e4b4b25f-8bda-4307-a7a5-b6d03a846ad7\" (UID: \"e4b4b25f-8bda-4307-a7a5-b6d03a846ad7\") "
Sep 15 12:00:03.645111 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.644991    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7871a402-06d0-4678-bd1e-1cdda46dd153-plugins-config-volume\") pod \"7871a402-06d0-4678-bd1e-1cdda46dd153\" (UID: \"7871a402-06d0-4678-bd1e-1cdda46dd153\") "
Sep 15 12:00:03.645111 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.645027    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz84r\" (UniqueName: \"kubernetes.io/projected/7871a402-06d0-4678-bd1e-1cdda46dd153-kube-api-access-tz84r\") pod \"7871a402-06d0-4678-bd1e-1cdda46dd153\" (UID: \"7871a402-06d0-4678-bd1e-1cdda46dd153\") "
Sep 15 12:00:03.645300 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.645265    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b4b25f-8bda-4307-a7a5-b6d03a846ad7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4b4b25f-8bda-4307-a7a5-b6d03a846ad7" (UID: "e4b4b25f-8bda-4307-a7a5-b6d03a846ad7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:00:03.645469 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.645449    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4b25f-8bda-4307-a7a5-b6d03a846ad7-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:03.645715 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.645680    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7871a402-06d0-4678-bd1e-1cdda46dd153-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7871a402-06d0-4678-bd1e-1cdda46dd153" (UID: "7871a402-06d0-4678-bd1e-1cdda46dd153"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:00:03.649538 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.649507    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b4b25f-8bda-4307-a7a5-b6d03a846ad7-kube-api-access-8598c" (OuterVolumeSpecName: "kube-api-access-8598c") pod "e4b4b25f-8bda-4307-a7a5-b6d03a846ad7" (UID: "e4b4b25f-8bda-4307-a7a5-b6d03a846ad7"). InnerVolumeSpecName "kube-api-access-8598c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:00:03.652129 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.651649    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7871a402-06d0-4678-bd1e-1cdda46dd153-kube-api-access-tz84r" (OuterVolumeSpecName: "kube-api-access-tz84r") pod "7871a402-06d0-4678-bd1e-1cdda46dd153" (UID: "7871a402-06d0-4678-bd1e-1cdda46dd153"). InnerVolumeSpecName "kube-api-access-tz84r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:00:03.690850 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.690821    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6dca39-29824560-wkqvn"
Sep 15 12:00:03.719119 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.719087    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ec8230-29824560-rnwkq"
Sep 15 12:00:03.733744 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.733715    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-29d630-29824560-wpv5n"
Sep 15 12:00:03.734162 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.734131    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1965e277-a7ba-4044-b8fe-b969650c849a" path="/var/lib/kubelet/pods/1965e277-a7ba-4044-b8fe-b969650c849a/volumes"
Sep 15 12:00:03.742512 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.742436    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds"
Sep 15 12:00:03.746147 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.746083    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8598c\" (UniqueName: \"kubernetes.io/projected/e4b4b25f-8bda-4307-a7a5-b6d03a846ad7-kube-api-access-8598c\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:03.746147 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.746110    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7871a402-06d0-4678-bd1e-1cdda46dd153-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:03.746147 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.746124    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz84r\" (UniqueName: \"kubernetes.io/projected/7871a402-06d0-4678-bd1e-1cdda46dd153-kube-api-access-tz84r\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:03.746794 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:03.746772    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac76e5-29824560-fjvmh"
Sep 15 12:00:04.414258 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.414231    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"
Sep 15 12:00:04.414438 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.414306    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"
Sep 15 12:00:04.502447 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.502368    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"
Sep 15 12:00:04.502447 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.502405    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q" event={"ID":"7871a402-06d0-4678-bd1e-1cdda46dd153","Type":"ContainerDied","Data":"ad28eccc97213f84cfe40f04dfa13a78c652bfd0488d996630c1fb6257f29d01"}
Sep 15 12:00:04.502956 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.502478    2580 scope.go:117] "RemoveContainer" containerID="6f9e413166760100ccfc55302c2f5fc0c18b065be1876d0a22c08e0163fb8a9a"
Sep 15 12:00:04.503858 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.503802    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac76e5-29824560-fjvmh"
Sep 15 12:00:04.503858 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.503831    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac76e5-29824560-fjvmh" event={"ID":"a6e97acc-8531-4a9a-b068-7425fc3a4434","Type":"ContainerDied","Data":"858e59edbeae8707acbfce98f5a498f9d913198fbf8009e0bd663f359babc1af"}
Sep 15 12:00:04.503997 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.503931    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="858e59edbeae8707acbfce98f5a498f9d913198fbf8009e0bd663f359babc1af"
Sep 15 12:00:04.505454 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.505437    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6dca39-29824560-wkqvn"
Sep 15 12:00:04.505555 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.505479    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6dca39-29824560-wkqvn" event={"ID":"c54ccce0-837b-4b9b-867d-5c9d45e33719","Type":"ContainerDied","Data":"a0e3cbd407974f18080139c4a59d7f9a94c8ad9fa207e504359e1d28fe6332fa"}
Sep 15 12:00:04.505555 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.505516    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0e3cbd407974f18080139c4a59d7f9a94c8ad9fa207e504359e1d28fe6332fa"
Sep 15 12:00:04.507630 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.507604    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ec8230-29824560-rnwkq"
Sep 15 12:00:04.507741 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.507714    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ec8230-29824560-rnwkq" event={"ID":"d52dc0ba-2fc3-44ba-b964-071011b0e912","Type":"ContainerDied","Data":"d0e0b03c807769347e382fe1525bdbd37ee2916967e9fb69189099672c678c63"}
Sep 15 12:00:04.507818 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.507746    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0e0b03c807769347e382fe1525bdbd37ee2916967e9fb69189099672c678c63"
Sep 15 12:00:04.509543 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.509526    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"
Sep 15 12:00:04.509672 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.509553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq" event={"ID":"e4b4b25f-8bda-4307-a7a5-b6d03a846ad7","Type":"ContainerDied","Data":"c49466853e1d47cae0c4ba823da7739ae30bb26bdcf33e9d814aff67a07f46fc"}
Sep 15 12:00:04.516498 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.516472    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds"
Sep 15 12:00:04.516498 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.516490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds" event={"ID":"9419a91e-0dc3-45b2-b3d6-0a9f0c7846a0","Type":"ContainerDied","Data":"f3322150b1324bb6a42b9897d4cba13fc91267195d0a100ac12b21fdae50bc0e"}
Sep 15 12:00:04.516642 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.516530    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3322150b1324bb6a42b9897d4cba13fc91267195d0a100ac12b21fdae50bc0e"
Sep 15 12:00:04.518355 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.518337    2580 scope.go:117] "RemoveContainer" containerID="82f03b86614251775906a220a0c687cbf15ec7a2d5e50a899c551c27080447d0"
Sep 15 12:00:04.518628 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.518611    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-29d630-29824560-wpv5n"
Sep 15 12:00:04.518707 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.518625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-29d630-29824560-wpv5n" event={"ID":"67547c19-52b2-4be3-8c21-7aa2253b647b","Type":"ContainerDied","Data":"80b4a1ee7a1ab142c91c0f46fee8876c64d213697ad8f9109ef097b433ecfd1d"}
Sep 15 12:00:04.518707 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.518658    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80b4a1ee7a1ab142c91c0f46fee8876c64d213697ad8f9109ef097b433ecfd1d"
Sep 15 12:00:04.526933 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.526893    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"]
Sep 15 12:00:04.530989 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.530966    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-97d02d-76c48cc98f-ss25q"]
Sep 15 12:00:04.565281 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.565254    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"]
Sep 15 12:00:04.574627 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.574602    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-97d02d-6bb9f4985d-qhzfq"]
Sep 15 12:00:04.585028 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.585004    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds"]
Sep 15 12:00:04.588291 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:04.588271    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-97d02d-29824560-kqdds"]
Sep 15 12:00:05.732144 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:05.732112    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7871a402-06d0-4678-bd1e-1cdda46dd153" path="/var/lib/kubelet/pods/7871a402-06d0-4678-bd1e-1cdda46dd153/volumes"
Sep 15 12:00:05.732499 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:05.732401    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9419a91e-0dc3-45b2-b3d6-0a9f0c7846a0" path="/var/lib/kubelet/pods/9419a91e-0dc3-45b2-b3d6-0a9f0c7846a0/volumes"
Sep 15 12:00:05.732690 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:05.732677    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4b4b25f-8bda-4307-a7a5-b6d03a846ad7" path="/var/lib/kubelet/pods/e4b4b25f-8bda-4307-a7a5-b6d03a846ad7/volumes"
Sep 15 12:00:08.743530 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:08.743494    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"]
Sep 15 12:00:19.381921 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.381866    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"]
Sep 15 12:00:19.382431 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382364    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6e97acc-8531-4a9a-b068-7425fc3a4434" containerName="cleanup"
Sep 15 12:00:19.382431 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382390    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6e97acc-8531-4a9a-b068-7425fc3a4434" containerName="cleanup"
Sep 15 12:00:19.382431 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382402    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52dc0ba-2fc3-44ba-b964-071011b0e912" containerName="cleanup"
Sep 15 12:00:19.382431 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382410    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52dc0ba-2fc3-44ba-b964-071011b0e912" containerName="cleanup"
Sep 15 12:00:19.382431 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382424    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b4b25f-8bda-4307-a7a5-b6d03a846ad7" containerName="payload-pre-processing"
Sep 15 12:00:19.382431 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382432    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b4b25f-8bda-4307-a7a5-b6d03a846ad7" containerName="payload-pre-processing"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382447    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9419a91e-0dc3-45b2-b3d6-0a9f0c7846a0" containerName="cleanup"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382454    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9419a91e-0dc3-45b2-b3d6-0a9f0c7846a0" containerName="cleanup"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382462    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7871a402-06d0-4678-bd1e-1cdda46dd153" containerName="payload-processing"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382468    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7871a402-06d0-4678-bd1e-1cdda46dd153" containerName="payload-processing"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382476    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c54ccce0-837b-4b9b-867d-5c9d45e33719" containerName="cleanup"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382482    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c54ccce0-837b-4b9b-867d-5c9d45e33719" containerName="cleanup"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382491    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67547c19-52b2-4be3-8c21-7aa2253b647b" containerName="cleanup"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382496    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="67547c19-52b2-4be3-8c21-7aa2253b647b" containerName="cleanup"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382604    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7871a402-06d0-4678-bd1e-1cdda46dd153" containerName="payload-processing"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382613    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9419a91e-0dc3-45b2-b3d6-0a9f0c7846a0" containerName="cleanup"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382622    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="67547c19-52b2-4be3-8c21-7aa2253b647b" containerName="cleanup"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382629    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c54ccce0-837b-4b9b-867d-5c9d45e33719" containerName="cleanup"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382636    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b4b25f-8bda-4307-a7a5-b6d03a846ad7" containerName="payload-pre-processing"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382642    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6e97acc-8531-4a9a-b068-7425fc3a4434" containerName="cleanup"
Sep 15 12:00:19.382807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.382646    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52dc0ba-2fc3-44ba-b964-071011b0e912" containerName="cleanup"
Sep 15 12:00:19.387184 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.387166    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.389837 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.389814    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-29d630\"/\"distinct-w0-29d630-kserve-self-signed-certs\""
Sep 15 12:00:19.408369 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.408341    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"]
Sep 15 12:00:19.507410 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.507378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-model-cache\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.507564 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.507459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e2c432-664d-437e-ab86-d91e2a0b3be5-tls-certs\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.507564 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.507490    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-dshm\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.507673 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.507558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-home\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.507673 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.507650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvtl4\" (UniqueName: \"kubernetes.io/projected/86e2c432-664d-437e-ab86-d91e2a0b3be5-kube-api-access-jvtl4\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.507740 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.507700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-tmp-dir\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.609076 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.609040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-model-cache\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.609269 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.609128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e2c432-664d-437e-ab86-d91e2a0b3be5-tls-certs\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.609269 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.609178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-dshm\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.609269 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.609224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-home\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.609269 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.609259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtl4\" (UniqueName: \"kubernetes.io/projected/86e2c432-664d-437e-ab86-d91e2a0b3be5-kube-api-access-jvtl4\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.609482 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.609304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-tmp-dir\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.609482 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.609443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-model-cache\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.609644 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.609624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-tmp-dir\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.609708 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.609651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-home\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.611659 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.611633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-dshm\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.611773 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.611711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e2c432-664d-437e-ab86-d91e2a0b3be5-tls-certs\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.621968 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.621945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvtl4\" (UniqueName: \"kubernetes.io/projected/86e2c432-664d-437e-ab86-d91e2a0b3be5-kube-api-access-jvtl4\") pod \"distinct-w0-29d630-kserve-857ddc94b7-nhrbz\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.696457 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.696426    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:19.981408 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:19.981381    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"]
Sep 15 12:00:19.981950 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:00:19.981928    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86e2c432_664d_437e_ab86_d91e2a0b3be5.slice/crio-ae12e284faed9b98e1b7d407a4f9ffbd751285a02cd6e5fd42e33ad3f550ca7e WatchSource:0}: Error finding container ae12e284faed9b98e1b7d407a4f9ffbd751285a02cd6e5fd42e33ad3f550ca7e: Status 404 returned error can't find the container with id ae12e284faed9b98e1b7d407a4f9ffbd751285a02cd6e5fd42e33ad3f550ca7e
Sep 15 12:00:20.624859 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:20.624824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz" event={"ID":"86e2c432-664d-437e-ab86-d91e2a0b3be5","Type":"ContainerStarted","Data":"7f12ef8f0955d31e5896f7087bfff8aa31f1cb0ce8ca0081c64fe2bf0ffca80a"}
Sep 15 12:00:20.625227 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:20.624890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz" event={"ID":"86e2c432-664d-437e-ab86-d91e2a0b3be5","Type":"ContainerStarted","Data":"ae12e284faed9b98e1b7d407a4f9ffbd751285a02cd6e5fd42e33ad3f550ca7e"}
Sep 15 12:00:20.645898 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:20.645833    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz" podStartSLOduration=1.645818839 podStartE2EDuration="1.645818839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:20.643840202 +0000 UTC m=+1309.424411593" watchObservedRunningTime="2026-09-15 12:00:20.645818839 +0000 UTC m=+1309.426390228"
Sep 15 12:00:23.763113 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:23.763071    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl" podUID="9f8e7e20-1e84-4b10-9a8c-26b00dae7eda" containerName="main" containerID="cri-o://4ef5b609ced0c4ba46793330d9a0a305ea74be7c12f06987a2fd19f9d405be7e" gracePeriod=45
Sep 15 12:00:24.038954 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.038924    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 12:00:24.153863 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.153834    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-model-cache\") pod \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") "
Sep 15 12:00:24.153863 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.153885    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-home\") pod \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") "
Sep 15 12:00:24.154117 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.153921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-tmp-dir\") pod \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") "
Sep 15 12:00:24.154117 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.153983    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmxpq\" (UniqueName: \"kubernetes.io/projected/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-kube-api-access-tmxpq\") pod \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") "
Sep 15 12:00:24.154117 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.154051    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-dshm\") pod \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") "
Sep 15 12:00:24.154117 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.154082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-tls-certs\") pod \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\" (UID: \"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda\") "
Sep 15 12:00:24.154117 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.154093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f8e7e20-1e84-4b10-9a8c-26b00dae7eda" (UID: "9f8e7e20-1e84-4b10-9a8c-26b00dae7eda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:00:24.154495 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.154167    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-home" (OuterVolumeSpecName: "home") pod "9f8e7e20-1e84-4b10-9a8c-26b00dae7eda" (UID: "9f8e7e20-1e84-4b10-9a8c-26b00dae7eda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:00:24.154495 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.154184    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f8e7e20-1e84-4b10-9a8c-26b00dae7eda" (UID: "9f8e7e20-1e84-4b10-9a8c-26b00dae7eda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:00:24.154604 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.154503    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:24.154604 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.154525    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:24.154604 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.154538    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:24.156696 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.156668    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-dshm" (OuterVolumeSpecName: "dshm") pod "9f8e7e20-1e84-4b10-9a8c-26b00dae7eda" (UID: "9f8e7e20-1e84-4b10-9a8c-26b00dae7eda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:00:24.156820 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.156785    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-kube-api-access-tmxpq" (OuterVolumeSpecName: "kube-api-access-tmxpq") pod "9f8e7e20-1e84-4b10-9a8c-26b00dae7eda" (UID: "9f8e7e20-1e84-4b10-9a8c-26b00dae7eda"). InnerVolumeSpecName "kube-api-access-tmxpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:00:24.157012 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.156992    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f8e7e20-1e84-4b10-9a8c-26b00dae7eda" (UID: "9f8e7e20-1e84-4b10-9a8c-26b00dae7eda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:00:24.257704 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.257612    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmxpq\" (UniqueName: \"kubernetes.io/projected/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-kube-api-access-tmxpq\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:24.257704 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.257660    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:24.257704 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.257676    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:24.653983 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.653942    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f8e7e20-1e84-4b10-9a8c-26b00dae7eda" containerID="4ef5b609ced0c4ba46793330d9a0a305ea74be7c12f06987a2fd19f9d405be7e" exitCode=0
Sep 15 12:00:24.654162 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.654000    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"
Sep 15 12:00:24.654162 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.654039    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl" event={"ID":"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda","Type":"ContainerDied","Data":"4ef5b609ced0c4ba46793330d9a0a305ea74be7c12f06987a2fd19f9d405be7e"}
Sep 15 12:00:24.654162 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.654095    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl" event={"ID":"9f8e7e20-1e84-4b10-9a8c-26b00dae7eda","Type":"ContainerDied","Data":"aa8ad614f7728feed7a6f6f8db5fc9297525a9c1208a96172f70c69e6283ac4b"}
Sep 15 12:00:24.654162 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.654121    2580 scope.go:117] "RemoveContainer" containerID="4ef5b609ced0c4ba46793330d9a0a305ea74be7c12f06987a2fd19f9d405be7e"
Sep 15 12:00:24.665555 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.665535    2580 scope.go:117] "RemoveContainer" containerID="4ef5b609ced0c4ba46793330d9a0a305ea74be7c12f06987a2fd19f9d405be7e"
Sep 15 12:00:24.665882 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:00:24.665847    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef5b609ced0c4ba46793330d9a0a305ea74be7c12f06987a2fd19f9d405be7e\": container with ID starting with 4ef5b609ced0c4ba46793330d9a0a305ea74be7c12f06987a2fd19f9d405be7e not found: ID does not exist" containerID="4ef5b609ced0c4ba46793330d9a0a305ea74be7c12f06987a2fd19f9d405be7e"
Sep 15 12:00:24.665975 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.665891    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef5b609ced0c4ba46793330d9a0a305ea74be7c12f06987a2fd19f9d405be7e"} err="failed to get container status \"4ef5b609ced0c4ba46793330d9a0a305ea74be7c12f06987a2fd19f9d405be7e\": rpc error: code = NotFound desc = could not find container \"4ef5b609ced0c4ba46793330d9a0a305ea74be7c12f06987a2fd19f9d405be7e\": container with ID starting with 4ef5b609ced0c4ba46793330d9a0a305ea74be7c12f06987a2fd19f9d405be7e not found: ID does not exist"
Sep 15 12:00:24.680181 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.680070    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"]
Sep 15 12:00:24.685348 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:24.685314    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-97d02d/test-model-97d02d-kserve-54bb8d5dcc-jz8rl"]
Sep 15 12:00:25.732982 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:25.732946    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f8e7e20-1e84-4b10-9a8c-26b00dae7eda" path="/var/lib/kubelet/pods/9f8e7e20-1e84-4b10-9a8c-26b00dae7eda/volumes"
Sep 15 12:00:26.652773 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:26.652730    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"]
Sep 15 12:00:29.697090 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:29.697057    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:29.697516 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:29.697104    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:29.709598 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:29.709567    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:30.711194 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:30.711160    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:00:31.837787 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:31.837753    2580 scope.go:117] "RemoveContainer" containerID="5b7df8ea8bb63407d30a71cdbebec17e9cb1c2c1a547f0ab02a6ba237a1a5e3a"
Sep 15 12:00:32.265633 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.265593    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"]
Sep 15 12:00:32.266018 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.266004    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f8e7e20-1e84-4b10-9a8c-26b00dae7eda" containerName="main"
Sep 15 12:00:32.266073 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.266019    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f8e7e20-1e84-4b10-9a8c-26b00dae7eda" containerName="main"
Sep 15 12:00:32.266110 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.266090    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f8e7e20-1e84-4b10-9a8c-26b00dae7eda" containerName="main"
Sep 15 12:00:32.270440 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.270425    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.273290 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.273268    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac76e5\"/\"e2e-conflict-b-6fb44257-kserve-self-signed-certs\""
Sep 15 12:00:32.286964 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.286941    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"]
Sep 15 12:00:32.347321 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.347294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-dshm\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.347461 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.347334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-tmp-dir\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.347461 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.347360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-model-cache\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.347461 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.347434    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-home\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.347600 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.347483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aceb058-207f-4255-9186-6f92cbea7478-tls-certs\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.347600 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.347501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm6xp\" (UniqueName: \"kubernetes.io/projected/0aceb058-207f-4255-9186-6f92cbea7478-kube-api-access-gm6xp\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.448838 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.448800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aceb058-207f-4255-9186-6f92cbea7478-tls-certs\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.448838 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.448837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6xp\" (UniqueName: \"kubernetes.io/projected/0aceb058-207f-4255-9186-6f92cbea7478-kube-api-access-gm6xp\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.449074 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.449039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-dshm\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.449113 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.449101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-tmp-dir\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.449167 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.449152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-model-cache\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.449243 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.449222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-home\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.449423 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.449395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-tmp-dir\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.449572 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.449552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-home\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.449572 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.449567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-model-cache\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.451258 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.451235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-dshm\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.451453 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.451436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aceb058-207f-4255-9186-6f92cbea7478-tls-certs\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.458322 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.458303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm6xp\" (UniqueName: \"kubernetes.io/projected/0aceb058-207f-4255-9186-6f92cbea7478-kube-api-access-gm6xp\") pod \"e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.580108 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.580017    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:32.857631 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:32.857571    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"]
Sep 15 12:00:32.859787 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:00:32.859758    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0aceb058_207f_4255_9186_6f92cbea7478.slice/crio-e985add996535731f9fb6577e8e5d4ec18064e7e40f3d9795935a58763b6c4b7 WatchSource:0}: Error finding container e985add996535731f9fb6577e8e5d4ec18064e7e40f3d9795935a58763b6c4b7: Status 404 returned error can't find the container with id e985add996535731f9fb6577e8e5d4ec18064e7e40f3d9795935a58763b6c4b7
Sep 15 12:00:33.720371 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:33.720329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7" event={"ID":"0aceb058-207f-4255-9186-6f92cbea7478","Type":"ContainerStarted","Data":"0654475a86a9f3da83968d3998b870c6f7a276a470fb48a4105fb668e43d76f7"}
Sep 15 12:00:33.720582 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:33.720384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7" event={"ID":"0aceb058-207f-4255-9186-6f92cbea7478","Type":"ContainerStarted","Data":"e985add996535731f9fb6577e8e5d4ec18064e7e40f3d9795935a58763b6c4b7"}
Sep 15 12:00:33.740625 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:33.740578    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7" podStartSLOduration=1.740563938 podStartE2EDuration="1.740563938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:33.737635071 +0000 UTC m=+1322.518206543" watchObservedRunningTime="2026-09-15 12:00:33.740563938 +0000 UTC m=+1322.521135361"
Sep 15 12:00:41.061638 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.061602    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"]
Sep 15 12:00:41.065954 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.065933    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.068334 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.068314    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cc097cc5\"/\"default-dockercfg-dbvvh\""
Sep 15 12:00:41.068679 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.068664    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cc097cc5\"/\"kube-root-ca.crt\""
Sep 15 12:00:41.069286 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.069265    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cc097cc5\"/\"e2e-lifecycle-model-cc097cc5-kserve-self-signed-certs\""
Sep 15 12:00:41.069374 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.069290    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cc097cc5\"/\"openshift-service-ca.crt\""
Sep 15 12:00:41.074130 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.074105    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"]
Sep 15 12:00:41.139186 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.139150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcj8f\" (UniqueName: \"kubernetes.io/projected/18366a98-9b18-464c-b427-a4b8ff749fe6-kube-api-access-hcj8f\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.139186 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.139191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-dshm\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.139381 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.139212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18366a98-9b18-464c-b427-a4b8ff749fe6-tls-certs\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.139381 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.139276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-home\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.139381 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.139359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-tmp-dir\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.139489 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.139407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-model-cache\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.240231 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.240196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-model-cache\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.240412 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.240258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcj8f\" (UniqueName: \"kubernetes.io/projected/18366a98-9b18-464c-b427-a4b8ff749fe6-kube-api-access-hcj8f\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.240412 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.240282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-dshm\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.240412 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.240300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18366a98-9b18-464c-b427-a4b8ff749fe6-tls-certs\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.240412 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.240323    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-home\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.240412 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.240392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-tmp-dir\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.240697 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.240675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-model-cache\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.240745 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.240723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-tmp-dir\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.240816 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.240790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-home\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.242614 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.242591    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-dshm\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.242762 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.242746    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18366a98-9b18-464c-b427-a4b8ff749fe6-tls-certs\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.248791 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.248768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcj8f\" (UniqueName: \"kubernetes.io/projected/18366a98-9b18-464c-b427-a4b8ff749fe6-kube-api-access-hcj8f\") pod \"e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") " pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.377983 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.377908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:41.651738 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.651709    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"]
Sep 15 12:00:41.652564 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:00:41.652536    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18366a98_9b18_464c_b427_a4b8ff749fe6.slice/crio-4e244443b79c040ff14703e32c39ddf24e02e9c5d76cc843e749bd9793b8bd7c WatchSource:0}: Error finding container 4e244443b79c040ff14703e32c39ddf24e02e9c5d76cc843e749bd9793b8bd7c: Status 404 returned error can't find the container with id 4e244443b79c040ff14703e32c39ddf24e02e9c5d76cc843e749bd9793b8bd7c
Sep 15 12:00:41.706301 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.706262    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5" podUID="a3273265-dd2e-420f-af85-e1bcea64eed6" containerName="main" containerID="cri-o://8a54eac684cdc8e6b263a9d4cc0ad25c5e1d8f33d819baa8cf025dfc3442e8ef" gracePeriod=45
Sep 15 12:00:41.771943 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.771910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg" event={"ID":"18366a98-9b18-464c-b427-a4b8ff749fe6","Type":"ContainerStarted","Data":"4e244443b79c040ff14703e32c39ddf24e02e9c5d76cc843e749bd9793b8bd7c"}
Sep 15 12:00:41.950774 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:41.950749    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 12:00:42.048562 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.048533    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-tmp-dir\") pod \"a3273265-dd2e-420f-af85-e1bcea64eed6\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") "
Sep 15 12:00:42.048727 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.048591    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-home\") pod \"a3273265-dd2e-420f-af85-e1bcea64eed6\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") "
Sep 15 12:00:42.048727 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.048631    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-dshm\") pod \"a3273265-dd2e-420f-af85-e1bcea64eed6\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") "
Sep 15 12:00:42.048727 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.048667    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3273265-dd2e-420f-af85-e1bcea64eed6-tls-certs\") pod \"a3273265-dd2e-420f-af85-e1bcea64eed6\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") "
Sep 15 12:00:42.048727 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.048711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvq45\" (UniqueName: \"kubernetes.io/projected/a3273265-dd2e-420f-af85-e1bcea64eed6-kube-api-access-tvq45\") pod \"a3273265-dd2e-420f-af85-e1bcea64eed6\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") "
Sep 15 12:00:42.049000 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.048737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-model-cache\") pod \"a3273265-dd2e-420f-af85-e1bcea64eed6\" (UID: \"a3273265-dd2e-420f-af85-e1bcea64eed6\") "
Sep 15 12:00:42.049000 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.048854    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3273265-dd2e-420f-af85-e1bcea64eed6" (UID: "a3273265-dd2e-420f-af85-e1bcea64eed6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:00:42.049000 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.048962    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-home" (OuterVolumeSpecName: "home") pod "a3273265-dd2e-420f-af85-e1bcea64eed6" (UID: "a3273265-dd2e-420f-af85-e1bcea64eed6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:00:42.049146 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.049094    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3273265-dd2e-420f-af85-e1bcea64eed6" (UID: "a3273265-dd2e-420f-af85-e1bcea64eed6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:00:42.049243 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.049220    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:42.049243 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.049239    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:42.049432 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.049252    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:42.050943 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.050920    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-dshm" (OuterVolumeSpecName: "dshm") pod "a3273265-dd2e-420f-af85-e1bcea64eed6" (UID: "a3273265-dd2e-420f-af85-e1bcea64eed6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:00:42.051050 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.050983    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3273265-dd2e-420f-af85-e1bcea64eed6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3273265-dd2e-420f-af85-e1bcea64eed6" (UID: "a3273265-dd2e-420f-af85-e1bcea64eed6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:00:42.051050 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.051013    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3273265-dd2e-420f-af85-e1bcea64eed6-kube-api-access-tvq45" (OuterVolumeSpecName: "kube-api-access-tvq45") pod "a3273265-dd2e-420f-af85-e1bcea64eed6" (UID: "a3273265-dd2e-420f-af85-e1bcea64eed6"). InnerVolumeSpecName "kube-api-access-tvq45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:00:42.150159 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.150126    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3273265-dd2e-420f-af85-e1bcea64eed6-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:42.150159 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.150154    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3273265-dd2e-420f-af85-e1bcea64eed6-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:42.150159 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.150163    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvq45\" (UniqueName: \"kubernetes.io/projected/a3273265-dd2e-420f-af85-e1bcea64eed6-kube-api-access-tvq45\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:42.580726 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.580640    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:42.580726 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.580696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:42.592865 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.592838    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:42.779730 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.779696    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg" event={"ID":"18366a98-9b18-464c-b427-a4b8ff749fe6","Type":"ContainerStarted","Data":"70ddc27dc259faa4f30d4833287304522b18c51f1b6e8e23a76128b41a0263ef"}
Sep 15 12:00:42.781067 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.781040    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3273265-dd2e-420f-af85-e1bcea64eed6" containerID="8a54eac684cdc8e6b263a9d4cc0ad25c5e1d8f33d819baa8cf025dfc3442e8ef" exitCode=0
Sep 15 12:00:42.781161 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.781081    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"
Sep 15 12:00:42.781161 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.781125    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5" event={"ID":"a3273265-dd2e-420f-af85-e1bcea64eed6","Type":"ContainerDied","Data":"8a54eac684cdc8e6b263a9d4cc0ad25c5e1d8f33d819baa8cf025dfc3442e8ef"}
Sep 15 12:00:42.781240 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.781162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5" event={"ID":"a3273265-dd2e-420f-af85-e1bcea64eed6","Type":"ContainerDied","Data":"6bee6aa65804394f2fa0dd8f63ced71faf0580e74351a5cba019659444705f72"}
Sep 15 12:00:42.781240 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.781184    2580 scope.go:117] "RemoveContainer" containerID="8a54eac684cdc8e6b263a9d4cc0ad25c5e1d8f33d819baa8cf025dfc3442e8ef"
Sep 15 12:00:42.792631 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.792606    2580 scope.go:117] "RemoveContainer" containerID="8a54eac684cdc8e6b263a9d4cc0ad25c5e1d8f33d819baa8cf025dfc3442e8ef"
Sep 15 12:00:42.792908 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:00:42.792865    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a54eac684cdc8e6b263a9d4cc0ad25c5e1d8f33d819baa8cf025dfc3442e8ef\": container with ID starting with 8a54eac684cdc8e6b263a9d4cc0ad25c5e1d8f33d819baa8cf025dfc3442e8ef not found: ID does not exist" containerID="8a54eac684cdc8e6b263a9d4cc0ad25c5e1d8f33d819baa8cf025dfc3442e8ef"
Sep 15 12:00:42.793005 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.792919    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a54eac684cdc8e6b263a9d4cc0ad25c5e1d8f33d819baa8cf025dfc3442e8ef"} err="failed to get container status \"8a54eac684cdc8e6b263a9d4cc0ad25c5e1d8f33d819baa8cf025dfc3442e8ef\": rpc error: code = NotFound desc = could not find container \"8a54eac684cdc8e6b263a9d4cc0ad25c5e1d8f33d819baa8cf025dfc3442e8ef\": container with ID starting with 8a54eac684cdc8e6b263a9d4cc0ad25c5e1d8f33d819baa8cf025dfc3442e8ef not found: ID does not exist"
Sep 15 12:00:42.795212 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.795192    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:00:42.798657 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.798619    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg" podStartSLOduration=1.798607246 podStartE2EDuration="1.798607246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:42.796902282 +0000 UTC m=+1331.577473671" watchObservedRunningTime="2026-09-15 12:00:42.798607246 +0000 UTC m=+1331.579178699"
Sep 15 12:00:42.831432 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.831372    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"]
Sep 15 12:00:42.838594 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:42.838566    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e9bc87/test-model-e9bc87-kserve-75c4b495f7-5jkt5"]
Sep 15 12:00:43.731775 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:43.731743    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3273265-dd2e-420f-af85-e1bcea64eed6" path="/var/lib/kubelet/pods/a3273265-dd2e-420f-af85-e1bcea64eed6/volumes"
Sep 15 12:00:46.866295 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:46.866261    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6"]
Sep 15 12:00:46.866663 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:46.866643    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3273265-dd2e-420f-af85-e1bcea64eed6" containerName="main"
Sep 15 12:00:46.866663 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:46.866656    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3273265-dd2e-420f-af85-e1bcea64eed6" containerName="main"
Sep 15 12:00:46.866743 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:46.866735    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3273265-dd2e-420f-af85-e1bcea64eed6" containerName="main"
Sep 15 12:00:46.871208 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:46.871189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6"
Sep 15 12:00:46.874281 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:46.874258    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:00:46.875125 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:46.875106    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:00:46.879580 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:46.879559    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6"]
Sep 15 12:00:46.998641 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:46.998603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7f9cf1-239f-4a4b-9559-92071f96d1e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6\" (UID: \"ba7f9cf1-239f-4a4b-9559-92071f96d1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6"
Sep 15 12:00:47.100173 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:47.100139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7f9cf1-239f-4a4b-9559-92071f96d1e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6\" (UID: \"ba7f9cf1-239f-4a4b-9559-92071f96d1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6"
Sep 15 12:00:47.100766 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:47.100747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7f9cf1-239f-4a4b-9559-92071f96d1e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6\" (UID: \"ba7f9cf1-239f-4a4b-9559-92071f96d1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6"
Sep 15 12:00:47.184064 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:47.184032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6"
Sep 15 12:00:47.457989 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:47.457960    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6"]
Sep 15 12:00:47.459368 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:00:47.459337    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7f9cf1_239f_4a4b_9559_92071f96d1e4.slice/crio-4fe8d865f88beb63f3bc768e62176ed9a5b258d44d28656643822ecd93af4394 WatchSource:0}: Error finding container 4fe8d865f88beb63f3bc768e62176ed9a5b258d44d28656643822ecd93af4394: Status 404 returned error can't find the container with id 4fe8d865f88beb63f3bc768e62176ed9a5b258d44d28656643822ecd93af4394
Sep 15 12:00:47.813261 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:47.813229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6" event={"ID":"ba7f9cf1-239f-4a4b-9559-92071f96d1e4","Type":"ContainerStarted","Data":"f8e42c2c4c559139007eb6b034b19d8f516e46229e2f4bf38fffbc984fb78c24"}
Sep 15 12:00:47.813394 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:47.813281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6" event={"ID":"ba7f9cf1-239f-4a4b-9559-92071f96d1e4","Type":"ContainerStarted","Data":"4fe8d865f88beb63f3bc768e62176ed9a5b258d44d28656643822ecd93af4394"}
Sep 15 12:00:47.829436 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:47.829386    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6" podStartSLOduration=1.829370841 podStartE2EDuration="1.829370841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:47.828598728 +0000 UTC m=+1336.609170116" watchObservedRunningTime="2026-09-15 12:00:47.829370841 +0000 UTC m=+1336.609942229"
Sep 15 12:00:48.820761 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:48.820726    2580 generic.go:356] "Generic (PLEG): container finished" podID="ba7f9cf1-239f-4a4b-9559-92071f96d1e4" containerID="f8e42c2c4c559139007eb6b034b19d8f516e46229e2f4bf38fffbc984fb78c24" exitCode=0
Sep 15 12:00:48.821218 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:48.820778    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6" event={"ID":"ba7f9cf1-239f-4a4b-9559-92071f96d1e4","Type":"ContainerDied","Data":"f8e42c2c4c559139007eb6b034b19d8f516e46229e2f4bf38fffbc984fb78c24"}
Sep 15 12:00:49.949807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:49.949781    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6"
Sep 15 12:00:50.029319 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:50.029293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7f9cf1-239f-4a4b-9559-92071f96d1e4-maas-api-service-ca\") pod \"ba7f9cf1-239f-4a4b-9559-92071f96d1e4\" (UID: \"ba7f9cf1-239f-4a4b-9559-92071f96d1e4\") "
Sep 15 12:00:50.029630 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:50.029607    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7f9cf1-239f-4a4b-9559-92071f96d1e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba7f9cf1-239f-4a4b-9559-92071f96d1e4" (UID: "ba7f9cf1-239f-4a4b-9559-92071f96d1e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:00:50.131008 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:50.130925    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7f9cf1-239f-4a4b-9559-92071f96d1e4-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:50.835146 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:50.835116    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6"
Sep 15 12:00:50.835319 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:50.835155    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6" event={"ID":"ba7f9cf1-239f-4a4b-9559-92071f96d1e4","Type":"ContainerDied","Data":"4fe8d865f88beb63f3bc768e62176ed9a5b258d44d28656643822ecd93af4394"}
Sep 15 12:00:50.835319 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:50.835201    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fe8d865f88beb63f3bc768e62176ed9a5b258d44d28656643822ecd93af4394"
Sep 15 12:00:51.378077 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:51.378041    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:51.378531 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:51.378090    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:51.390645 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:51.390620    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:51.852386 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:51.852354    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:00:57.643367 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:57.643288    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"]
Sep 15 12:00:57.643945 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:57.643529    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2" podUID="ee50cb3f-198a-41e3-876f-1ed5db6fdeae" containerName="payload-pre-processing" containerID="cri-o://bb62dac143fc8f743090fdd2a2e8cf16c6a430d08e28a91d53c1096e31fdd81e" gracePeriod=30
Sep 15 12:00:57.821318 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:57.821255    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6"]
Sep 15 12:00:57.824505 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:57.824475    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e9bc87-64sr6"]
Sep 15 12:00:57.879289 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:57.879256    2580 generic.go:356] "Generic (PLEG): container finished" podID="ee50cb3f-198a-41e3-876f-1ed5db6fdeae" containerID="bb62dac143fc8f743090fdd2a2e8cf16c6a430d08e28a91d53c1096e31fdd81e" exitCode=0
Sep 15 12:00:57.879408 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:57.879324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2" event={"ID":"ee50cb3f-198a-41e3-876f-1ed5db6fdeae","Type":"ContainerDied","Data":"bb62dac143fc8f743090fdd2a2e8cf16c6a430d08e28a91d53c1096e31fdd81e"}
Sep 15 12:00:57.879408 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:57.879386    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2" event={"ID":"ee50cb3f-198a-41e3-876f-1ed5db6fdeae","Type":"ContainerDied","Data":"e456706533baba9e9509515d4d81cfceda7c83f83d479279bfe71d5f5f8daaaa"}
Sep 15 12:00:57.879492 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:57.879416    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e456706533baba9e9509515d4d81cfceda7c83f83d479279bfe71d5f5f8daaaa"
Sep 15 12:00:57.893663 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:57.893608    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"
Sep 15 12:00:58.009622 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:58.009586    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smjpz\" (UniqueName: \"kubernetes.io/projected/ee50cb3f-198a-41e3-876f-1ed5db6fdeae-kube-api-access-smjpz\") pod \"ee50cb3f-198a-41e3-876f-1ed5db6fdeae\" (UID: \"ee50cb3f-198a-41e3-876f-1ed5db6fdeae\") "
Sep 15 12:00:58.009801 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:58.009674    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee50cb3f-198a-41e3-876f-1ed5db6fdeae-plugins-config-volume\") pod \"ee50cb3f-198a-41e3-876f-1ed5db6fdeae\" (UID: \"ee50cb3f-198a-41e3-876f-1ed5db6fdeae\") "
Sep 15 12:00:58.010117 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:58.010094    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee50cb3f-198a-41e3-876f-1ed5db6fdeae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee50cb3f-198a-41e3-876f-1ed5db6fdeae" (UID: "ee50cb3f-198a-41e3-876f-1ed5db6fdeae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:00:58.011703 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:58.011672    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee50cb3f-198a-41e3-876f-1ed5db6fdeae-kube-api-access-smjpz" (OuterVolumeSpecName: "kube-api-access-smjpz") pod "ee50cb3f-198a-41e3-876f-1ed5db6fdeae" (UID: "ee50cb3f-198a-41e3-876f-1ed5db6fdeae"). InnerVolumeSpecName "kube-api-access-smjpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:00:58.110616 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:58.110573    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smjpz\" (UniqueName: \"kubernetes.io/projected/ee50cb3f-198a-41e3-876f-1ed5db6fdeae-kube-api-access-smjpz\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:58.110616 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:58.110608    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee50cb3f-198a-41e3-876f-1ed5db6fdeae-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:00:58.885566 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:58.885529    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"
Sep 15 12:00:58.909189 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:58.909161    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"]
Sep 15 12:00:58.923035 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:58.923003    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e9bc87-6d4556dcf4-7lpl2"]
Sep 15 12:00:59.731130 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:59.731097    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba7f9cf1-239f-4a4b-9559-92071f96d1e4" path="/var/lib/kubelet/pods/ba7f9cf1-239f-4a4b-9559-92071f96d1e4/volumes"
Sep 15 12:00:59.731463 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:00:59.731449    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee50cb3f-198a-41e3-876f-1ed5db6fdeae" path="/var/lib/kubelet/pods/ee50cb3f-198a-41e3-876f-1ed5db6fdeae/volumes"
Sep 15 12:01:31.876548 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:31.876518    2580 scope.go:117] "RemoveContainer" containerID="bdf7af673862def60dd8df3170deb4dbce7fbe93e06ade9f8ae73396b5d5655f"
Sep 15 12:01:31.885425 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:31.885404    2580 scope.go:117] "RemoveContainer" containerID="bb62dac143fc8f743090fdd2a2e8cf16c6a430d08e28a91d53c1096e31fdd81e"
Sep 15 12:01:31.893694 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:31.893677    2580 scope.go:117] "RemoveContainer" containerID="b43b80d6cc2a9bfa3b2625722d5fcdc784210632518842d3debc6727fe0adac8"
Sep 15 12:01:47.581243 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.581203    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl"]
Sep 15 12:01:47.581807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.581770    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba7f9cf1-239f-4a4b-9559-92071f96d1e4" containerName="revoke-keys"
Sep 15 12:01:47.581807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.581791    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba7f9cf1-239f-4a4b-9559-92071f96d1e4" containerName="revoke-keys"
Sep 15 12:01:47.581973 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.581810    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee50cb3f-198a-41e3-876f-1ed5db6fdeae" containerName="payload-pre-processing"
Sep 15 12:01:47.581973 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.581819    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee50cb3f-198a-41e3-876f-1ed5db6fdeae" containerName="payload-pre-processing"
Sep 15 12:01:47.582115 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.581988    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee50cb3f-198a-41e3-876f-1ed5db6fdeae" containerName="payload-pre-processing"
Sep 15 12:01:47.582115 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.582006    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba7f9cf1-239f-4a4b-9559-92071f96d1e4" containerName="revoke-keys"
Sep 15 12:01:47.586671 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.586649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl"
Sep 15 12:01:47.591594 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.591573    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:01:47.591716 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.591603    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:01:47.598318 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.598293    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl"]
Sep 15 12:01:47.712241 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.712210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d912b9-e164-4320-90f1-2697ea109b56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl\" (UID: \"69d912b9-e164-4320-90f1-2697ea109b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl"
Sep 15 12:01:47.813187 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.813147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d912b9-e164-4320-90f1-2697ea109b56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl\" (UID: \"69d912b9-e164-4320-90f1-2697ea109b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl"
Sep 15 12:01:47.813749 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.813730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d912b9-e164-4320-90f1-2697ea109b56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl\" (UID: \"69d912b9-e164-4320-90f1-2697ea109b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl"
Sep 15 12:01:47.897590 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:47.897512    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl"
Sep 15 12:01:48.176084 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:48.176058    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl"]
Sep 15 12:01:48.177724 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:01:48.177694    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d912b9_e164_4320_90f1_2697ea109b56.slice/crio-5c6f5cbb8157cf31314b5aba8618addc2341dd653738f81512c3455836849d08 WatchSource:0}: Error finding container 5c6f5cbb8157cf31314b5aba8618addc2341dd653738f81512c3455836849d08: Status 404 returned error can't find the container with id 5c6f5cbb8157cf31314b5aba8618addc2341dd653738f81512c3455836849d08
Sep 15 12:01:48.189855 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:48.189828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl" event={"ID":"69d912b9-e164-4320-90f1-2697ea109b56","Type":"ContainerStarted","Data":"5c6f5cbb8157cf31314b5aba8618addc2341dd653738f81512c3455836849d08"}
Sep 15 12:01:49.198615 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:49.198579    2580 generic.go:356] "Generic (PLEG): container finished" podID="69d912b9-e164-4320-90f1-2697ea109b56" containerID="0c1beb714879262ee86d0bf771f9302e69bc1790c376bdb19b43a29aa10f2426" exitCode=0
Sep 15 12:01:49.199086 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:49.198645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl" event={"ID":"69d912b9-e164-4320-90f1-2697ea109b56","Type":"ContainerDied","Data":"0c1beb714879262ee86d0bf771f9302e69bc1790c376bdb19b43a29aa10f2426"}
Sep 15 12:01:50.344785 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:50.344762    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl"
Sep 15 12:01:50.439735 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:50.439707    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d912b9-e164-4320-90f1-2697ea109b56-maas-api-service-ca\") pod \"69d912b9-e164-4320-90f1-2697ea109b56\" (UID: \"69d912b9-e164-4320-90f1-2697ea109b56\") "
Sep 15 12:01:50.440090 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:50.440069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d912b9-e164-4320-90f1-2697ea109b56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69d912b9-e164-4320-90f1-2697ea109b56" (UID: "69d912b9-e164-4320-90f1-2697ea109b56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:01:50.540501 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:50.540426    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d912b9-e164-4320-90f1-2697ea109b56-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:01:51.211960 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:51.211925    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl"
Sep 15 12:01:51.211960 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:51.211941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl" event={"ID":"69d912b9-e164-4320-90f1-2697ea109b56","Type":"ContainerDied","Data":"5c6f5cbb8157cf31314b5aba8618addc2341dd653738f81512c3455836849d08"}
Sep 15 12:01:51.212214 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:01:51.211989    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c6f5cbb8157cf31314b5aba8618addc2341dd653738f81512c3455836849d08"
Sep 15 12:02:03.528276 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.528239    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"]
Sep 15 12:02:03.528900 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.528484    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs" podUID="d1b26bff-dc0e-4f72-9951-3d2b638aaedd" containerName="payload-processing" containerID="cri-o://c0c2346f051b9ceec805a8ba4103237a9da5cca312a5a3f9bf1e24f43d3e8402" gracePeriod=30
Sep 15 12:02:03.537804 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.537767    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"]
Sep 15 12:02:03.539823 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.538526    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96" podUID="14d868b5-01cd-410a-87da-908ec2aacc64" containerName="payload-pre-processing" containerID="cri-o://261353cf556d84e692cd11646e3b586900e59107527d697263f9367936b7935e" gracePeriod=30
Sep 15 12:02:03.732684 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.732652    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl"]
Sep 15 12:02:03.732837 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.732699    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc097cc5-ckpbl"]
Sep 15 12:02:03.808207 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.808181    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"
Sep 15 12:02:03.834745 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.834710    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"
Sep 15 12:02:03.975826 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.975785    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b26bff-dc0e-4f72-9951-3d2b638aaedd-plugins-config-volume\") pod \"d1b26bff-dc0e-4f72-9951-3d2b638aaedd\" (UID: \"d1b26bff-dc0e-4f72-9951-3d2b638aaedd\") "
Sep 15 12:02:03.976062 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.975978    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95lnh\" (UniqueName: \"kubernetes.io/projected/14d868b5-01cd-410a-87da-908ec2aacc64-kube-api-access-95lnh\") pod \"14d868b5-01cd-410a-87da-908ec2aacc64\" (UID: \"14d868b5-01cd-410a-87da-908ec2aacc64\") "
Sep 15 12:02:03.976062 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.976045    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blrk4\" (UniqueName: \"kubernetes.io/projected/d1b26bff-dc0e-4f72-9951-3d2b638aaedd-kube-api-access-blrk4\") pod \"d1b26bff-dc0e-4f72-9951-3d2b638aaedd\" (UID: \"d1b26bff-dc0e-4f72-9951-3d2b638aaedd\") "
Sep 15 12:02:03.976184 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.976074    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d868b5-01cd-410a-87da-908ec2aacc64-plugins-config-volume\") pod \"14d868b5-01cd-410a-87da-908ec2aacc64\" (UID: \"14d868b5-01cd-410a-87da-908ec2aacc64\") "
Sep 15 12:02:03.976308 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.976277    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b26bff-dc0e-4f72-9951-3d2b638aaedd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1b26bff-dc0e-4f72-9951-3d2b638aaedd" (UID: "d1b26bff-dc0e-4f72-9951-3d2b638aaedd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:02:03.976496 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.976478    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b26bff-dc0e-4f72-9951-3d2b638aaedd-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:03.976560 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.976524    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d868b5-01cd-410a-87da-908ec2aacc64-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14d868b5-01cd-410a-87da-908ec2aacc64" (UID: "14d868b5-01cd-410a-87da-908ec2aacc64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:02:03.978150 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.978129    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b26bff-dc0e-4f72-9951-3d2b638aaedd-kube-api-access-blrk4" (OuterVolumeSpecName: "kube-api-access-blrk4") pod "d1b26bff-dc0e-4f72-9951-3d2b638aaedd" (UID: "d1b26bff-dc0e-4f72-9951-3d2b638aaedd"). InnerVolumeSpecName "kube-api-access-blrk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:02:03.978227 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:03.978166    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d868b5-01cd-410a-87da-908ec2aacc64-kube-api-access-95lnh" (OuterVolumeSpecName: "kube-api-access-95lnh") pod "14d868b5-01cd-410a-87da-908ec2aacc64" (UID: "14d868b5-01cd-410a-87da-908ec2aacc64"). InnerVolumeSpecName "kube-api-access-95lnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:02:04.078013 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.077912    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blrk4\" (UniqueName: \"kubernetes.io/projected/d1b26bff-dc0e-4f72-9951-3d2b638aaedd-kube-api-access-blrk4\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:04.078013 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.077951    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d868b5-01cd-410a-87da-908ec2aacc64-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:04.078013 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.077964    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95lnh\" (UniqueName: \"kubernetes.io/projected/14d868b5-01cd-410a-87da-908ec2aacc64-kube-api-access-95lnh\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:04.294139 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.294105    2580 generic.go:356] "Generic (PLEG): container finished" podID="14d868b5-01cd-410a-87da-908ec2aacc64" containerID="261353cf556d84e692cd11646e3b586900e59107527d697263f9367936b7935e" exitCode=0
Sep 15 12:02:04.294323 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.294169    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"
Sep 15 12:02:04.294323 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.294192    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96" event={"ID":"14d868b5-01cd-410a-87da-908ec2aacc64","Type":"ContainerDied","Data":"261353cf556d84e692cd11646e3b586900e59107527d697263f9367936b7935e"}
Sep 15 12:02:04.294323 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.294273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96" event={"ID":"14d868b5-01cd-410a-87da-908ec2aacc64","Type":"ContainerDied","Data":"ad92585fb572f88f4e3e6f8724e16454b7d5920549851ef9a8b6896e1ad6e1ec"}
Sep 15 12:02:04.294323 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.294295    2580 scope.go:117] "RemoveContainer" containerID="261353cf556d84e692cd11646e3b586900e59107527d697263f9367936b7935e"
Sep 15 12:02:04.295991 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.295965    2580 generic.go:356] "Generic (PLEG): container finished" podID="d1b26bff-dc0e-4f72-9951-3d2b638aaedd" containerID="c0c2346f051b9ceec805a8ba4103237a9da5cca312a5a3f9bf1e24f43d3e8402" exitCode=0
Sep 15 12:02:04.296081 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.296015    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"
Sep 15 12:02:04.296081 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.296043    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs" event={"ID":"d1b26bff-dc0e-4f72-9951-3d2b638aaedd","Type":"ContainerDied","Data":"c0c2346f051b9ceec805a8ba4103237a9da5cca312a5a3f9bf1e24f43d3e8402"}
Sep 15 12:02:04.296171 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.296086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs" event={"ID":"d1b26bff-dc0e-4f72-9951-3d2b638aaedd","Type":"ContainerDied","Data":"b031515db5b04b1d46536025f4bff05ce0137be60fc5195494be1c506046f7b0"}
Sep 15 12:02:04.303747 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.303712    2580 scope.go:117] "RemoveContainer" containerID="261353cf556d84e692cd11646e3b586900e59107527d697263f9367936b7935e"
Sep 15 12:02:04.304091 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:02:04.304066    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261353cf556d84e692cd11646e3b586900e59107527d697263f9367936b7935e\": container with ID starting with 261353cf556d84e692cd11646e3b586900e59107527d697263f9367936b7935e not found: ID does not exist" containerID="261353cf556d84e692cd11646e3b586900e59107527d697263f9367936b7935e"
Sep 15 12:02:04.304181 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.304096    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261353cf556d84e692cd11646e3b586900e59107527d697263f9367936b7935e"} err="failed to get container status \"261353cf556d84e692cd11646e3b586900e59107527d697263f9367936b7935e\": rpc error: code = NotFound desc = could not find container \"261353cf556d84e692cd11646e3b586900e59107527d697263f9367936b7935e\": container with ID starting with 261353cf556d84e692cd11646e3b586900e59107527d697263f9367936b7935e not found: ID does not exist"
Sep 15 12:02:04.304181 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.304112    2580 scope.go:117] "RemoveContainer" containerID="c0c2346f051b9ceec805a8ba4103237a9da5cca312a5a3f9bf1e24f43d3e8402"
Sep 15 12:02:04.312867 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.312852    2580 scope.go:117] "RemoveContainer" containerID="c0c2346f051b9ceec805a8ba4103237a9da5cca312a5a3f9bf1e24f43d3e8402"
Sep 15 12:02:04.313154 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:02:04.313136    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c2346f051b9ceec805a8ba4103237a9da5cca312a5a3f9bf1e24f43d3e8402\": container with ID starting with c0c2346f051b9ceec805a8ba4103237a9da5cca312a5a3f9bf1e24f43d3e8402 not found: ID does not exist" containerID="c0c2346f051b9ceec805a8ba4103237a9da5cca312a5a3f9bf1e24f43d3e8402"
Sep 15 12:02:04.313198 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.313162    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c2346f051b9ceec805a8ba4103237a9da5cca312a5a3f9bf1e24f43d3e8402"} err="failed to get container status \"c0c2346f051b9ceec805a8ba4103237a9da5cca312a5a3f9bf1e24f43d3e8402\": rpc error: code = NotFound desc = could not find container \"c0c2346f051b9ceec805a8ba4103237a9da5cca312a5a3f9bf1e24f43d3e8402\": container with ID starting with c0c2346f051b9ceec805a8ba4103237a9da5cca312a5a3f9bf1e24f43d3e8402 not found: ID does not exist"
Sep 15 12:02:04.321765 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.321737    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"]
Sep 15 12:02:04.325912 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.325890    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cc097cc5-5ff94cfcf7-847xs"]
Sep 15 12:02:04.337773 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.337721    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"]
Sep 15 12:02:04.342400 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:04.342377    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cc097cc5-757dff96bb-b9j96"]
Sep 15 12:02:05.731768 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:05.731735    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14d868b5-01cd-410a-87da-908ec2aacc64" path="/var/lib/kubelet/pods/14d868b5-01cd-410a-87da-908ec2aacc64/volumes"
Sep 15 12:02:05.732155 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:05.732141    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69d912b9-e164-4320-90f1-2697ea109b56" path="/var/lib/kubelet/pods/69d912b9-e164-4320-90f1-2697ea109b56/volumes"
Sep 15 12:02:05.732513 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:05.732498    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1b26bff-dc0e-4f72-9951-3d2b638aaedd" path="/var/lib/kubelet/pods/d1b26bff-dc0e-4f72-9951-3d2b638aaedd/volumes"
Sep 15 12:02:11.814514 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:11.814469    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"]
Sep 15 12:02:14.766691 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.766656    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"]
Sep 15 12:02:14.767228 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.767209    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d868b5-01cd-410a-87da-908ec2aacc64" containerName="payload-pre-processing"
Sep 15 12:02:14.767314 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.767231    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d868b5-01cd-410a-87da-908ec2aacc64" containerName="payload-pre-processing"
Sep 15 12:02:14.767314 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.767282    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1b26bff-dc0e-4f72-9951-3d2b638aaedd" containerName="payload-processing"
Sep 15 12:02:14.767314 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.767292    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1b26bff-dc0e-4f72-9951-3d2b638aaedd" containerName="payload-processing"
Sep 15 12:02:14.767480 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.767325    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d912b9-e164-4320-90f1-2697ea109b56" containerName="revoke-keys"
Sep 15 12:02:14.767480 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.767335    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d912b9-e164-4320-90f1-2697ea109b56" containerName="revoke-keys"
Sep 15 12:02:14.767480 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.767439    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1b26bff-dc0e-4f72-9951-3d2b638aaedd" containerName="payload-processing"
Sep 15 12:02:14.767480 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.767453    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="14d868b5-01cd-410a-87da-908ec2aacc64" containerName="payload-pre-processing"
Sep 15 12:02:14.767480 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.767465    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d912b9-e164-4320-90f1-2697ea109b56" containerName="revoke-keys"
Sep 15 12:02:14.780157 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.780128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.782514 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.782489    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"]
Sep 15 12:02:14.786179 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.786158    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-29d630\"/\"distinct-2-w0-29d630-kserve-self-signed-certs\""
Sep 15 12:02:14.876905 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.876850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-home\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.877107 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.876918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3399618-3c99-46fb-80aa-f18a1558ef5b-tls-certs\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.877107 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.876981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zplb5\" (UniqueName: \"kubernetes.io/projected/a3399618-3c99-46fb-80aa-f18a1558ef5b-kube-api-access-zplb5\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.877107 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.877022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-model-cache\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.877107 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.877093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-tmp-dir\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.877302 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.877154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-dshm\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.978316 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.978282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-tmp-dir\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.978515 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.978337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-dshm\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.978515 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.978422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-home\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.978515 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.978446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3399618-3c99-46fb-80aa-f18a1558ef5b-tls-certs\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.978515 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.978510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zplb5\" (UniqueName: \"kubernetes.io/projected/a3399618-3c99-46fb-80aa-f18a1558ef5b-kube-api-access-zplb5\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.978756 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.978560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-model-cache\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.978756 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.978694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-tmp-dir\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.978911 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.978809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-home\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.978986 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.978929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-model-cache\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.980690 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.980658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-dshm\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:14.980899 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:14.980853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3399618-3c99-46fb-80aa-f18a1558ef5b-tls-certs\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:15.001193 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:15.001169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zplb5\" (UniqueName: \"kubernetes.io/projected/a3399618-3c99-46fb-80aa-f18a1558ef5b-kube-api-access-zplb5\") pod \"distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") " pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:15.089766 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:15.089688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:15.370652 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:15.370620    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"]
Sep 15 12:02:15.372341 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:02:15.372315    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3399618_3c99_46fb_80aa_f18a1558ef5b.slice/crio-68dc78da6fc860e9c4783b54e81df75821ee3e5c8528eb1df30104c1951685a7 WatchSource:0}: Error finding container 68dc78da6fc860e9c4783b54e81df75821ee3e5c8528eb1df30104c1951685a7: Status 404 returned error can't find the container with id 68dc78da6fc860e9c4783b54e81df75821ee3e5c8528eb1df30104c1951685a7
Sep 15 12:02:16.370816 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:16.370780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f" event={"ID":"a3399618-3c99-46fb-80aa-f18a1558ef5b","Type":"ContainerStarted","Data":"20bc015aa867bf610e8eac493e6ae6067fd53944b564b47be0c8bd919dbd8f5a"}
Sep 15 12:02:16.371314 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:16.370826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f" event={"ID":"a3399618-3c99-46fb-80aa-f18a1558ef5b","Type":"ContainerStarted","Data":"68dc78da6fc860e9c4783b54e81df75821ee3e5c8528eb1df30104c1951685a7"}
Sep 15 12:02:16.392509 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:16.392457    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f" podStartSLOduration=2.392442073 podStartE2EDuration="2.392442073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:16.388801722 +0000 UTC m=+1425.169373110" watchObservedRunningTime="2026-09-15 12:02:16.392442073 +0000 UTC m=+1425.173013461"
Sep 15 12:02:17.268933 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:17.268866    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"]
Sep 15 12:02:25.089838 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:25.089761    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:25.089838 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:25.089807    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:25.102367 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:25.102345    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:25.445784 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:25.445753    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:02:26.833631 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:26.833567    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg" podUID="18366a98-9b18-464c-b427-a4b8ff749fe6" containerName="main" containerID="cri-o://70ddc27dc259faa4f30d4833287304522b18c51f1b6e8e23a76128b41a0263ef" gracePeriod=45
Sep 15 12:02:27.076091 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.076070    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:02:27.201589 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.201561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-tmp-dir\") pod \"18366a98-9b18-464c-b427-a4b8ff749fe6\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") "
Sep 15 12:02:27.201768 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.201611    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-home\") pod \"18366a98-9b18-464c-b427-a4b8ff749fe6\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") "
Sep 15 12:02:27.201768 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.201644    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcj8f\" (UniqueName: \"kubernetes.io/projected/18366a98-9b18-464c-b427-a4b8ff749fe6-kube-api-access-hcj8f\") pod \"18366a98-9b18-464c-b427-a4b8ff749fe6\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") "
Sep 15 12:02:27.201768 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.201700    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18366a98-9b18-464c-b427-a4b8ff749fe6-tls-certs\") pod \"18366a98-9b18-464c-b427-a4b8ff749fe6\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") "
Sep 15 12:02:27.201768 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.201720    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-dshm\") pod \"18366a98-9b18-464c-b427-a4b8ff749fe6\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") "
Sep 15 12:02:27.201768 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.201736    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-model-cache\") pod \"18366a98-9b18-464c-b427-a4b8ff749fe6\" (UID: \"18366a98-9b18-464c-b427-a4b8ff749fe6\") "
Sep 15 12:02:27.202075 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.201835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18366a98-9b18-464c-b427-a4b8ff749fe6" (UID: "18366a98-9b18-464c-b427-a4b8ff749fe6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:02:27.202075 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.201907    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-home" (OuterVolumeSpecName: "home") pod "18366a98-9b18-464c-b427-a4b8ff749fe6" (UID: "18366a98-9b18-464c-b427-a4b8ff749fe6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:02:27.202075 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.202052    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-model-cache" (OuterVolumeSpecName: "model-cache") pod "18366a98-9b18-464c-b427-a4b8ff749fe6" (UID: "18366a98-9b18-464c-b427-a4b8ff749fe6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:02:27.202180 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.202106    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:27.202180 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.202121    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:27.203923 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.203901    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18366a98-9b18-464c-b427-a4b8ff749fe6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18366a98-9b18-464c-b427-a4b8ff749fe6" (UID: "18366a98-9b18-464c-b427-a4b8ff749fe6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:02:27.204189 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.204169    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-dshm" (OuterVolumeSpecName: "dshm") pod "18366a98-9b18-464c-b427-a4b8ff749fe6" (UID: "18366a98-9b18-464c-b427-a4b8ff749fe6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:02:27.204257 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.204175    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18366a98-9b18-464c-b427-a4b8ff749fe6-kube-api-access-hcj8f" (OuterVolumeSpecName: "kube-api-access-hcj8f") pod "18366a98-9b18-464c-b427-a4b8ff749fe6" (UID: "18366a98-9b18-464c-b427-a4b8ff749fe6"). InnerVolumeSpecName "kube-api-access-hcj8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:02:27.303232 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.303184    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcj8f\" (UniqueName: \"kubernetes.io/projected/18366a98-9b18-464c-b427-a4b8ff749fe6-kube-api-access-hcj8f\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:27.303232 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.303226    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18366a98-9b18-464c-b427-a4b8ff749fe6-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:27.303232 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.303237    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:27.303232 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.303244    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18366a98-9b18-464c-b427-a4b8ff749fe6-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:27.448858 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.448818    2580 generic.go:356] "Generic (PLEG): container finished" podID="18366a98-9b18-464c-b427-a4b8ff749fe6" containerID="70ddc27dc259faa4f30d4833287304522b18c51f1b6e8e23a76128b41a0263ef" exitCode=0
Sep 15 12:02:27.448858 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.448857    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"
Sep 15 12:02:27.449094 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.448908    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg" event={"ID":"18366a98-9b18-464c-b427-a4b8ff749fe6","Type":"ContainerDied","Data":"70ddc27dc259faa4f30d4833287304522b18c51f1b6e8e23a76128b41a0263ef"}
Sep 15 12:02:27.449094 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.448950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg" event={"ID":"18366a98-9b18-464c-b427-a4b8ff749fe6","Type":"ContainerDied","Data":"4e244443b79c040ff14703e32c39ddf24e02e9c5d76cc843e749bd9793b8bd7c"}
Sep 15 12:02:27.449094 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.448968    2580 scope.go:117] "RemoveContainer" containerID="70ddc27dc259faa4f30d4833287304522b18c51f1b6e8e23a76128b41a0263ef"
Sep 15 12:02:27.465406 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.465387    2580 scope.go:117] "RemoveContainer" containerID="70ddc27dc259faa4f30d4833287304522b18c51f1b6e8e23a76128b41a0263ef"
Sep 15 12:02:27.465681 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:02:27.465663    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ddc27dc259faa4f30d4833287304522b18c51f1b6e8e23a76128b41a0263ef\": container with ID starting with 70ddc27dc259faa4f30d4833287304522b18c51f1b6e8e23a76128b41a0263ef not found: ID does not exist" containerID="70ddc27dc259faa4f30d4833287304522b18c51f1b6e8e23a76128b41a0263ef"
Sep 15 12:02:27.465730 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.465690    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ddc27dc259faa4f30d4833287304522b18c51f1b6e8e23a76128b41a0263ef"} err="failed to get container status \"70ddc27dc259faa4f30d4833287304522b18c51f1b6e8e23a76128b41a0263ef\": rpc error: code = NotFound desc = could not find container \"70ddc27dc259faa4f30d4833287304522b18c51f1b6e8e23a76128b41a0263ef\": container with ID starting with 70ddc27dc259faa4f30d4833287304522b18c51f1b6e8e23a76128b41a0263ef not found: ID does not exist"
Sep 15 12:02:27.477145 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.477118    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"]
Sep 15 12:02:27.479566 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.479547    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-cc097cc5/e2e-lifecycle-model-cc097cc5-kserve-76b856766c-lvxgg"]
Sep 15 12:02:27.731486 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:27.731405    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18366a98-9b18-464c-b427-a4b8ff749fe6" path="/var/lib/kubelet/pods/18366a98-9b18-464c-b427-a4b8ff749fe6/volumes"
Sep 15 12:02:31.930684 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:31.930653    2580 scope.go:117] "RemoveContainer" containerID="18b55350ae4731856bc67e5d13a796fb7590127026d24091df3671fbf320b451"
Sep 15 12:02:32.288923 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.288815    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7" podUID="0aceb058-207f-4255-9186-6f92cbea7478" containerName="main" containerID="cri-o://0654475a86a9f3da83968d3998b870c6f7a276a470fb48a4105fb668e43d76f7" gracePeriod=45
Sep 15 12:02:32.486653 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.486623    2580 generic.go:356] "Generic (PLEG): container finished" podID="0aceb058-207f-4255-9186-6f92cbea7478" containerID="0654475a86a9f3da83968d3998b870c6f7a276a470fb48a4105fb668e43d76f7" exitCode=0
Sep 15 12:02:32.486813 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.486706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7" event={"ID":"0aceb058-207f-4255-9186-6f92cbea7478","Type":"ContainerDied","Data":"0654475a86a9f3da83968d3998b870c6f7a276a470fb48a4105fb668e43d76f7"}
Sep 15 12:02:32.540127 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.540071    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:02:32.651292 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.651250    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aceb058-207f-4255-9186-6f92cbea7478-tls-certs\") pod \"0aceb058-207f-4255-9186-6f92cbea7478\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") "
Sep 15 12:02:32.651487 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.651361    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm6xp\" (UniqueName: \"kubernetes.io/projected/0aceb058-207f-4255-9186-6f92cbea7478-kube-api-access-gm6xp\") pod \"0aceb058-207f-4255-9186-6f92cbea7478\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") "
Sep 15 12:02:32.651487 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.651401    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-home\") pod \"0aceb058-207f-4255-9186-6f92cbea7478\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") "
Sep 15 12:02:32.651487 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.651422    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-model-cache\") pod \"0aceb058-207f-4255-9186-6f92cbea7478\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") "
Sep 15 12:02:32.651487 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.651466    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-tmp-dir\") pod \"0aceb058-207f-4255-9186-6f92cbea7478\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") "
Sep 15 12:02:32.651716 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.651527    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-dshm\") pod \"0aceb058-207f-4255-9186-6f92cbea7478\" (UID: \"0aceb058-207f-4255-9186-6f92cbea7478\") "
Sep 15 12:02:32.651716 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.651691    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-home" (OuterVolumeSpecName: "home") pod "0aceb058-207f-4255-9186-6f92cbea7478" (UID: "0aceb058-207f-4255-9186-6f92cbea7478"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:02:32.651833 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.651706    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-model-cache" (OuterVolumeSpecName: "model-cache") pod "0aceb058-207f-4255-9186-6f92cbea7478" (UID: "0aceb058-207f-4255-9186-6f92cbea7478"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:02:32.651833 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.651808    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0aceb058-207f-4255-9186-6f92cbea7478" (UID: "0aceb058-207f-4255-9186-6f92cbea7478"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:02:32.651974 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.651940    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:32.651974 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.651960    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:32.651974 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.651969    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:32.653698 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.653670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-dshm" (OuterVolumeSpecName: "dshm") pod "0aceb058-207f-4255-9186-6f92cbea7478" (UID: "0aceb058-207f-4255-9186-6f92cbea7478"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:02:32.653698 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.653669    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aceb058-207f-4255-9186-6f92cbea7478-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0aceb058-207f-4255-9186-6f92cbea7478" (UID: "0aceb058-207f-4255-9186-6f92cbea7478"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:02:32.653850 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.653737    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aceb058-207f-4255-9186-6f92cbea7478-kube-api-access-gm6xp" (OuterVolumeSpecName: "kube-api-access-gm6xp") pod "0aceb058-207f-4255-9186-6f92cbea7478" (UID: "0aceb058-207f-4255-9186-6f92cbea7478"). InnerVolumeSpecName "kube-api-access-gm6xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:02:32.753401 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.753361    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm6xp\" (UniqueName: \"kubernetes.io/projected/0aceb058-207f-4255-9186-6f92cbea7478-kube-api-access-gm6xp\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:32.753401 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.753393    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aceb058-207f-4255-9186-6f92cbea7478-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:32.753401 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:32.753405    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aceb058-207f-4255-9186-6f92cbea7478-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:02:33.494452 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:33.494419    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"
Sep 15 12:02:33.494452 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:33.494445    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7" event={"ID":"0aceb058-207f-4255-9186-6f92cbea7478","Type":"ContainerDied","Data":"e985add996535731f9fb6577e8e5d4ec18064e7e40f3d9795935a58763b6c4b7"}
Sep 15 12:02:33.495052 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:33.494499    2580 scope.go:117] "RemoveContainer" containerID="0654475a86a9f3da83968d3998b870c6f7a276a470fb48a4105fb668e43d76f7"
Sep 15 12:02:33.518324 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:33.518300    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"]
Sep 15 12:02:33.522526 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:33.522502    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-b-6fb44257-kserve-6687b787d7-bpjs7"]
Sep 15 12:02:33.731156 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:33.731120    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0aceb058-207f-4255-9186-6f92cbea7478" path="/var/lib/kubelet/pods/0aceb058-207f-4255-9186-6f92cbea7478/volumes"
Sep 15 12:02:39.472983 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.472949    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"]
Sep 15 12:02:39.473461 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.473315    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18366a98-9b18-464c-b427-a4b8ff749fe6" containerName="main"
Sep 15 12:02:39.473461 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.473327    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18366a98-9b18-464c-b427-a4b8ff749fe6" containerName="main"
Sep 15 12:02:39.473461 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.473335    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aceb058-207f-4255-9186-6f92cbea7478" containerName="main"
Sep 15 12:02:39.473461 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.473340    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aceb058-207f-4255-9186-6f92cbea7478" containerName="main"
Sep 15 12:02:39.473461 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.473399    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aceb058-207f-4255-9186-6f92cbea7478" containerName="main"
Sep 15 12:02:39.473461 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.473408    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18366a98-9b18-464c-b427-a4b8ff749fe6" containerName="main"
Sep 15 12:02:39.477698 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.477676    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.481690 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.481666    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b3936\"/\"openshift-service-ca.crt\""
Sep 15 12:02:39.481690 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.481686    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b3936\"/\"default-dockercfg-j2s5q\""
Sep 15 12:02:39.481884 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.481666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b3936\"/\"auth-test-model-2b3936-kserve-self-signed-certs\""
Sep 15 12:02:39.481884 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.481758    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b3936\"/\"kube-root-ca.crt\""
Sep 15 12:02:39.488321 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.488299    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"]
Sep 15 12:02:39.623919 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.623860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e72aa51-3b91-433a-b376-cc146f292559-tls-certs\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.624115 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.623941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-model-cache\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.624115 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.623958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-home\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.624115 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.623994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-tmp-dir\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.624115 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.624105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-dshm\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.624323 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.624205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhcvq\" (UniqueName: \"kubernetes.io/projected/3e72aa51-3b91-433a-b376-cc146f292559-kube-api-access-rhcvq\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.724854 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.724763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-tmp-dir\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.724854 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.724837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-dshm\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.725110 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.724929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhcvq\" (UniqueName: \"kubernetes.io/projected/3e72aa51-3b91-433a-b376-cc146f292559-kube-api-access-rhcvq\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.725110 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.724972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e72aa51-3b91-433a-b376-cc146f292559-tls-certs\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.725110 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.725017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-model-cache\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.725110 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.725040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-home\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.725310 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.725210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-tmp-dir\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.725373 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.725355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-home\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.725478 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.725459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-model-cache\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.727440 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.727417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e72aa51-3b91-433a-b376-cc146f292559-tls-certs\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.727575 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.727474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-dshm\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.736182 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.736162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhcvq\" (UniqueName: \"kubernetes.io/projected/3e72aa51-3b91-433a-b376-cc146f292559-kube-api-access-rhcvq\") pod \"auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:39.787759 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:39.787715    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:40.077588 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:40.077563    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"]
Sep 15 12:02:40.079756 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:02:40.079725    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e72aa51_3b91_433a_b376_cc146f292559.slice/crio-530b8473c8a70e10a74723ea1e9444a230dcd2cf65290f344ef027d09d19c711 WatchSource:0}: Error finding container 530b8473c8a70e10a74723ea1e9444a230dcd2cf65290f344ef027d09d19c711: Status 404 returned error can't find the container with id 530b8473c8a70e10a74723ea1e9444a230dcd2cf65290f344ef027d09d19c711
Sep 15 12:02:40.540060 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:40.540022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2" event={"ID":"3e72aa51-3b91-433a-b376-cc146f292559","Type":"ContainerStarted","Data":"415f765382a1a330df74d8cb8ae23a309665e67831c62dd032d74fcfb5e463ec"}
Sep 15 12:02:40.540416 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:40.540073    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2" event={"ID":"3e72aa51-3b91-433a-b376-cc146f292559","Type":"ContainerStarted","Data":"530b8473c8a70e10a74723ea1e9444a230dcd2cf65290f344ef027d09d19c711"}
Sep 15 12:02:40.562045 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:40.562000    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2" podStartSLOduration=1.561986701 podStartE2EDuration="1.561986701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:40.560683721 +0000 UTC m=+1449.341255108" watchObservedRunningTime="2026-09-15 12:02:40.561986701 +0000 UTC m=+1449.342558088"
Sep 15 12:02:49.788855 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:49.788817    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:49.789451 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:49.788867    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:49.801580 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:49.801553    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:02:50.609434 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:02:50.609404    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:03:08.264752 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.264708    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"]
Sep 15 12:03:08.268290 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.268267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.270329 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.270309    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac76e5\"/\"distinct-w4-ac76e5-kserve-self-signed-certs\""
Sep 15 12:03:08.277167 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.277143    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"]
Sep 15 12:03:08.390668 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.390635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-tmp-dir\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.390851 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.390690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-model-cache\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.390851 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.390715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm429\" (UniqueName: \"kubernetes.io/projected/df1da465-3751-4cf8-b8a8-e9faa35fd98f-kube-api-access-pm429\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.390851 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.390744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-dshm\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.390851 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.390779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-home\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.390851 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.390810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1da465-3751-4cf8-b8a8-e9faa35fd98f-tls-certs\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.492017 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.491982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-dshm\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.492196 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.492031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-home\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.492196 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.492063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1da465-3751-4cf8-b8a8-e9faa35fd98f-tls-certs\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.492196 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.492148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-tmp-dir\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.492330 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.492218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-model-cache\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.492330 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.492251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm429\" (UniqueName: \"kubernetes.io/projected/df1da465-3751-4cf8-b8a8-e9faa35fd98f-kube-api-access-pm429\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.492553 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.492526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-home\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.492636 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.492536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-tmp-dir\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.492696 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.492632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-model-cache\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.494914 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.494895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-dshm\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.495121 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.495105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1da465-3751-4cf8-b8a8-e9faa35fd98f-tls-certs\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.500827 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.500806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm429\" (UniqueName: \"kubernetes.io/projected/df1da465-3751-4cf8-b8a8-e9faa35fd98f-kube-api-access-pm429\") pod \"distinct-w4-ac76e5-kserve-77896b84dd-nkf2h\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.578905 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.578809    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:08.854249 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:08.854216    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"]
Sep 15 12:03:08.855985 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:03:08.855960    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf1da465_3751_4cf8_b8a8_e9faa35fd98f.slice/crio-8d44720587cd80eeaf4c6f42d457ccace77f178ec1810191436283462acda837 WatchSource:0}: Error finding container 8d44720587cd80eeaf4c6f42d457ccace77f178ec1810191436283462acda837: Status 404 returned error can't find the container with id 8d44720587cd80eeaf4c6f42d457ccace77f178ec1810191436283462acda837
Sep 15 12:03:09.711702 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:09.711663    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h" event={"ID":"df1da465-3751-4cf8-b8a8-e9faa35fd98f","Type":"ContainerStarted","Data":"ea946eaee32b0636911616d3e087917edfd7df38737b6d351d7fdc205dd06a03"}
Sep 15 12:03:09.712159 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:09.711725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h" event={"ID":"df1da465-3751-4cf8-b8a8-e9faa35fd98f","Type":"ContainerStarted","Data":"8d44720587cd80eeaf4c6f42d457ccace77f178ec1810191436283462acda837"}
Sep 15 12:03:09.734654 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:09.734600    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h" podStartSLOduration=1.734584807 podStartE2EDuration="1.734584807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:09.733193373 +0000 UTC m=+1478.513764762" watchObservedRunningTime="2026-09-15 12:03:09.734584807 +0000 UTC m=+1478.515156195"
Sep 15 12:03:18.579181 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:18.579141    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:18.579659 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:18.579204    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:18.591486 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:18.591459    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:03:18.776829 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:03:18.776800    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:04:21.257787 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.257752    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"]
Sep 15 12:04:21.261378 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.261356    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.263610 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.263583    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac76e5\"/\"distinct-2-w4-ac76e5-kserve-self-signed-certs\""
Sep 15 12:04:21.273703 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.273680    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"]
Sep 15 12:04:21.323216 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.323178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-dshm\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.323414 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.323228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-model-cache\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.323414 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.323250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf72h\" (UniqueName: \"kubernetes.io/projected/856006a0-813c-40e5-aca3-e6937e75a532-kube-api-access-lf72h\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.323571 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.323407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-home\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.323571 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.323495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/856006a0-813c-40e5-aca3-e6937e75a532-tls-certs\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.323571 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.323547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-tmp-dir\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.424418 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.424378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-home\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.424631 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.424428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/856006a0-813c-40e5-aca3-e6937e75a532-tls-certs\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.424631 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.424477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-tmp-dir\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.424631 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.424524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-dshm\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.424631 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.424563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-model-cache\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.424631 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.424588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf72h\" (UniqueName: \"kubernetes.io/projected/856006a0-813c-40e5-aca3-e6937e75a532-kube-api-access-lf72h\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.425010 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.424982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-tmp-dir\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.425128 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.425102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-model-cache\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.425194 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.425157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-home\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.426906 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.426881    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-dshm\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.427083 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.427066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/856006a0-813c-40e5-aca3-e6937e75a532-tls-certs\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.432684 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.432665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf72h\" (UniqueName: \"kubernetes.io/projected/856006a0-813c-40e5-aca3-e6937e75a532-kube-api-access-lf72h\") pod \"distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") " pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.575498 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.575411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:21.851445 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.851417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"]
Sep 15 12:04:21.853594 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:04:21.853570    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod856006a0_813c_40e5_aca3_e6937e75a532.slice/crio-04f2f176a2ac3b199b8557c8670e94cfd4d8d2ea6978ce3191cda17efbb1f963 WatchSource:0}: Error finding container 04f2f176a2ac3b199b8557c8670e94cfd4d8d2ea6978ce3191cda17efbb1f963: Status 404 returned error can't find the container with id 04f2f176a2ac3b199b8557c8670e94cfd4d8d2ea6978ce3191cda17efbb1f963
Sep 15 12:04:21.913983 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.913958    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"]
Sep 15 12:04:21.917285 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.917269    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"
Sep 15 12:04:21.919682 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.919663    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-25d1cb\""
Sep 15 12:04:21.933375 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.933343    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"]
Sep 15 12:04:21.949181 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.949150    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"]
Sep 15 12:04:21.953009 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.952988    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"
Sep 15 12:04:21.955761 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.955735    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-25d1cb-dockercfg-ckbc5\""
Sep 15 12:04:21.966094 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:21.966065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"]
Sep 15 12:04:22.031115 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.031077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b6566d-420f-416e-bdfe-84ac73a0e6d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt\" (UID: \"44b6566d-420f-416e-bdfe-84ac73a0e6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"
Sep 15 12:04:22.031299 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.031146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h7j9\" (UniqueName: \"kubernetes.io/projected/44b6566d-420f-416e-bdfe-84ac73a0e6d2-kube-api-access-8h7j9\") pod \"payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt\" (UID: \"44b6566d-420f-416e-bdfe-84ac73a0e6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"
Sep 15 12:04:22.031299 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.031204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kzvt\" (UniqueName: \"kubernetes.io/projected/49b32be3-73ad-4d16-9fb2-f85cd743797d-kube-api-access-9kzvt\") pod \"payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r\" (UID: \"49b32be3-73ad-4d16-9fb2-f85cd743797d\") " pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"
Sep 15 12:04:22.031299 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.031273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b32be3-73ad-4d16-9fb2-f85cd743797d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r\" (UID: \"49b32be3-73ad-4d16-9fb2-f85cd743797d\") " pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"
Sep 15 12:04:22.132192 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.132100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h7j9\" (UniqueName: \"kubernetes.io/projected/44b6566d-420f-416e-bdfe-84ac73a0e6d2-kube-api-access-8h7j9\") pod \"payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt\" (UID: \"44b6566d-420f-416e-bdfe-84ac73a0e6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"
Sep 15 12:04:22.132192 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.132173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kzvt\" (UniqueName: \"kubernetes.io/projected/49b32be3-73ad-4d16-9fb2-f85cd743797d-kube-api-access-9kzvt\") pod \"payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r\" (UID: \"49b32be3-73ad-4d16-9fb2-f85cd743797d\") " pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"
Sep 15 12:04:22.132443 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.132200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b32be3-73ad-4d16-9fb2-f85cd743797d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r\" (UID: \"49b32be3-73ad-4d16-9fb2-f85cd743797d\") " pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"
Sep 15 12:04:22.132443 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.132261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b6566d-420f-416e-bdfe-84ac73a0e6d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt\" (UID: \"44b6566d-420f-416e-bdfe-84ac73a0e6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"
Sep 15 12:04:22.132936 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.132916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b32be3-73ad-4d16-9fb2-f85cd743797d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r\" (UID: \"49b32be3-73ad-4d16-9fb2-f85cd743797d\") " pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"
Sep 15 12:04:22.132987 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.132962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b6566d-420f-416e-bdfe-84ac73a0e6d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt\" (UID: \"44b6566d-420f-416e-bdfe-84ac73a0e6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"
Sep 15 12:04:22.140713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.140690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kzvt\" (UniqueName: \"kubernetes.io/projected/49b32be3-73ad-4d16-9fb2-f85cd743797d-kube-api-access-9kzvt\") pod \"payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r\" (UID: \"49b32be3-73ad-4d16-9fb2-f85cd743797d\") " pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"
Sep 15 12:04:22.140815 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.140757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h7j9\" (UniqueName: \"kubernetes.io/projected/44b6566d-420f-416e-bdfe-84ac73a0e6d2-kube-api-access-8h7j9\") pod \"payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt\" (UID: \"44b6566d-420f-416e-bdfe-84ac73a0e6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"
Sep 15 12:04:22.154851 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.154824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4" event={"ID":"856006a0-813c-40e5-aca3-e6937e75a532","Type":"ContainerStarted","Data":"34d915be02f327e5169fe2ed67f4130b33c6f4ecb9c07b0772dc1b985eba3193"}
Sep 15 12:04:22.154967 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.154866    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4" event={"ID":"856006a0-813c-40e5-aca3-e6937e75a532","Type":"ContainerStarted","Data":"04f2f176a2ac3b199b8557c8670e94cfd4d8d2ea6978ce3191cda17efbb1f963"}
Sep 15 12:04:22.174552 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.174510    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4" podStartSLOduration=1.174495938 podStartE2EDuration="1.174495938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:04:22.17175766 +0000 UTC m=+1550.952329050" watchObservedRunningTime="2026-09-15 12:04:22.174495938 +0000 UTC m=+1550.955067327"
Sep 15 12:04:22.228063 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.228028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"
Sep 15 12:04:22.264595 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.264565    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"
Sep 15 12:04:22.503999 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.503966    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"]
Sep 15 12:04:22.504746 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:04:22.504717    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b6566d_420f_416e_bdfe_84ac73a0e6d2.slice/crio-b1b66771a335304db7c5b6ec13287b93eafc2c93314642d42eeea06c0fcf1126 WatchSource:0}: Error finding container b1b66771a335304db7c5b6ec13287b93eafc2c93314642d42eeea06c0fcf1126: Status 404 returned error can't find the container with id b1b66771a335304db7c5b6ec13287b93eafc2c93314642d42eeea06c0fcf1126
Sep 15 12:04:22.551739 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:22.551715    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"]
Sep 15 12:04:22.552519 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:04:22.552496    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b32be3_73ad_4d16_9fb2_f85cd743797d.slice/crio-1a2f52f2b22e97932e884510976ac05f6e52f46a21262f23967257c1b065d8a5 WatchSource:0}: Error finding container 1a2f52f2b22e97932e884510976ac05f6e52f46a21262f23967257c1b065d8a5: Status 404 returned error can't find the container with id 1a2f52f2b22e97932e884510976ac05f6e52f46a21262f23967257c1b065d8a5
Sep 15 12:04:23.163539 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:23.163491    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r" event={"ID":"49b32be3-73ad-4d16-9fb2-f85cd743797d","Type":"ContainerStarted","Data":"0cf7c5f1d3ab49de925aca99ef984f1bed5c209425b8e84ebb6aee3f25c9dc5c"}
Sep 15 12:04:23.163749 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:23.163592    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"
Sep 15 12:04:23.163749 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:23.163617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r" event={"ID":"49b32be3-73ad-4d16-9fb2-f85cd743797d","Type":"ContainerStarted","Data":"1a2f52f2b22e97932e884510976ac05f6e52f46a21262f23967257c1b065d8a5"}
Sep 15 12:04:23.165445 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:23.165409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt" event={"ID":"44b6566d-420f-416e-bdfe-84ac73a0e6d2","Type":"ContainerStarted","Data":"ec94d1c4e5ef130313efc74751500765d2550eed46f666a3ef635d0ef5ade3a6"}
Sep 15 12:04:23.165589 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:23.165480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt" event={"ID":"44b6566d-420f-416e-bdfe-84ac73a0e6d2","Type":"ContainerStarted","Data":"b1b66771a335304db7c5b6ec13287b93eafc2c93314642d42eeea06c0fcf1126"}
Sep 15 12:04:23.165683 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:23.165651    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"
Sep 15 12:04:23.181719 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:23.181676    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r" podStartSLOduration=2.181661662 podStartE2EDuration="2.181661662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:04:23.179523668 +0000 UTC m=+1551.960095077" watchObservedRunningTime="2026-09-15 12:04:23.181661662 +0000 UTC m=+1551.962233050"
Sep 15 12:04:23.197168 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:23.197123    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt" podStartSLOduration=2.197111595 podStartE2EDuration="2.197111595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:04:23.194888451 +0000 UTC m=+1551.975459833" watchObservedRunningTime="2026-09-15 12:04:23.197111595 +0000 UTC m=+1551.977683012"
Sep 15 12:04:31.576385 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:31.576338    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:31.576774 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:31.576398    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:31.589084 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:31.589054    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:31.978295 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:31.978259    2580 scope.go:117] "RemoveContainer" containerID="fee6de98ddd33b66861422c7f297cfbcda7e53a7526ec280b5a70af97a31cc8e"
Sep 15 12:04:31.987409 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:31.987391    2580 scope.go:117] "RemoveContainer" containerID="360c8239ad91e835bc74912f1bd270a79abaa0aa760123a7460529d2334538bf"
Sep 15 12:04:32.238370 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:32.238278    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:04:34.173387 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:34.173351    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"
Sep 15 12:04:34.173911 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:34.173441    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"
Sep 15 12:04:52.920220 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:52.920182    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"]
Sep 15 12:04:52.924154 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:52.924130    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"
Sep 15 12:04:52.927354 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:52.927336    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0848f9-dockercfg-wfd6c\""
Sep 15 12:04:52.927667 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:52.927652    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0848f9\""
Sep 15 12:04:52.938406 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:52.938377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4q26\" (UniqueName: \"kubernetes.io/projected/bfde6835-a1ea-4ea9-aad6-42aae307cfa7-kube-api-access-s4q26\") pod \"payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f\" (UID: \"bfde6835-a1ea-4ea9-aad6-42aae307cfa7\") " pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"
Sep 15 12:04:52.938520 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:52.938484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfde6835-a1ea-4ea9-aad6-42aae307cfa7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f\" (UID: \"bfde6835-a1ea-4ea9-aad6-42aae307cfa7\") " pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"
Sep 15 12:04:52.940717 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:52.940692    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"]
Sep 15 12:04:53.039609 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:53.039572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfde6835-a1ea-4ea9-aad6-42aae307cfa7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f\" (UID: \"bfde6835-a1ea-4ea9-aad6-42aae307cfa7\") " pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"
Sep 15 12:04:53.039793 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:53.039659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4q26\" (UniqueName: \"kubernetes.io/projected/bfde6835-a1ea-4ea9-aad6-42aae307cfa7-kube-api-access-s4q26\") pod \"payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f\" (UID: \"bfde6835-a1ea-4ea9-aad6-42aae307cfa7\") " pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"
Sep 15 12:04:53.040299 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:53.040277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfde6835-a1ea-4ea9-aad6-42aae307cfa7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f\" (UID: \"bfde6835-a1ea-4ea9-aad6-42aae307cfa7\") " pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"
Sep 15 12:04:53.050462 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:53.050445    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4q26\" (UniqueName: \"kubernetes.io/projected/bfde6835-a1ea-4ea9-aad6-42aae307cfa7-kube-api-access-s4q26\") pod \"payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f\" (UID: \"bfde6835-a1ea-4ea9-aad6-42aae307cfa7\") " pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"
Sep 15 12:04:53.234427 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:53.234348    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"
Sep 15 12:04:53.515325 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:53.515300    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"]
Sep 15 12:04:53.516582 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:04:53.516548    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfde6835_a1ea_4ea9_aad6_42aae307cfa7.slice/crio-dd143e114194199e214d7ee3f399516e6982f8134b2283b9a3d02b6d7df73d3d WatchSource:0}: Error finding container dd143e114194199e214d7ee3f399516e6982f8134b2283b9a3d02b6d7df73d3d: Status 404 returned error can't find the container with id dd143e114194199e214d7ee3f399516e6982f8134b2283b9a3d02b6d7df73d3d
Sep 15 12:04:54.362760 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:54.362721    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f" event={"ID":"bfde6835-a1ea-4ea9-aad6-42aae307cfa7","Type":"ContainerStarted","Data":"29ddb587594dc68799f1f413726494b9d9e4fd498cac0d03f088b025c941f01f"}
Sep 15 12:04:54.363165 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:54.362783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f" event={"ID":"bfde6835-a1ea-4ea9-aad6-42aae307cfa7","Type":"ContainerStarted","Data":"dd143e114194199e214d7ee3f399516e6982f8134b2283b9a3d02b6d7df73d3d"}
Sep 15 12:04:54.363165 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:54.362813    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"
Sep 15 12:04:54.390645 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:04:54.390596    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f" podStartSLOduration=2.390581432 podStartE2EDuration="2.390581432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:04:54.38808587 +0000 UTC m=+1583.168657259" watchObservedRunningTime="2026-09-15 12:04:54.390581432 +0000 UTC m=+1583.171152822"
Sep 15 12:05:05.379491 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:05.379457    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"
Sep 15 12:05:07.937772 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:07.937736    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w"]
Sep 15 12:05:07.941587 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:07.941565    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w"
Sep 15 12:05:07.943839 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:07.943816    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:05:07.943959 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:07.943887    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:05:07.949762 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:07.949394    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w"]
Sep 15 12:05:08.084396 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:08.084349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9eb100d-e6e8-48df-8957-44704b70f0ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w\" (UID: \"e9eb100d-e6e8-48df-8957-44704b70f0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w"
Sep 15 12:05:08.185345 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:08.185314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9eb100d-e6e8-48df-8957-44704b70f0ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w\" (UID: \"e9eb100d-e6e8-48df-8957-44704b70f0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w"
Sep 15 12:05:08.186015 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:08.185997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9eb100d-e6e8-48df-8957-44704b70f0ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w\" (UID: \"e9eb100d-e6e8-48df-8957-44704b70f0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w"
Sep 15 12:05:08.254205 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:08.254140    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w"
Sep 15 12:05:08.527772 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:08.527743    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w"]
Sep 15 12:05:08.529236 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:05:08.529204    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9eb100d_e6e8_48df_8957_44704b70f0ae.slice/crio-6796475e853f3bf55d4fe1b22eedacf81147213cfc7dee3d5b97a660d33f9a28 WatchSource:0}: Error finding container 6796475e853f3bf55d4fe1b22eedacf81147213cfc7dee3d5b97a660d33f9a28: Status 404 returned error can't find the container with id 6796475e853f3bf55d4fe1b22eedacf81147213cfc7dee3d5b97a660d33f9a28
Sep 15 12:05:09.460050 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:09.459962    2580 generic.go:356] "Generic (PLEG): container finished" podID="e9eb100d-e6e8-48df-8957-44704b70f0ae" containerID="a886c78f782be8dbb4287b43ca259992ad47a3c9011a22d92d2d5437654a7c74" exitCode=0
Sep 15 12:05:09.460050 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:09.460021    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w" event={"ID":"e9eb100d-e6e8-48df-8957-44704b70f0ae","Type":"ContainerDied","Data":"a886c78f782be8dbb4287b43ca259992ad47a3c9011a22d92d2d5437654a7c74"}
Sep 15 12:05:09.460436 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:09.460057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w" event={"ID":"e9eb100d-e6e8-48df-8957-44704b70f0ae","Type":"ContainerStarted","Data":"6796475e853f3bf55d4fe1b22eedacf81147213cfc7dee3d5b97a660d33f9a28"}
Sep 15 12:05:10.606950 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:10.606924    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w"
Sep 15 12:05:10.710118 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:10.710089    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9eb100d-e6e8-48df-8957-44704b70f0ae-maas-api-service-ca\") pod \"e9eb100d-e6e8-48df-8957-44704b70f0ae\" (UID: \"e9eb100d-e6e8-48df-8957-44704b70f0ae\") "
Sep 15 12:05:10.710430 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:10.710410    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9eb100d-e6e8-48df-8957-44704b70f0ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9eb100d-e6e8-48df-8957-44704b70f0ae" (UID: "e9eb100d-e6e8-48df-8957-44704b70f0ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:05:10.811905 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:10.811793    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9eb100d-e6e8-48df-8957-44704b70f0ae-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:05:11.474455 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:11.474425    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w"
Sep 15 12:05:11.474630 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:11.474460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w" event={"ID":"e9eb100d-e6e8-48df-8957-44704b70f0ae","Type":"ContainerDied","Data":"6796475e853f3bf55d4fe1b22eedacf81147213cfc7dee3d5b97a660d33f9a28"}
Sep 15 12:05:11.474630 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:11.474508    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6796475e853f3bf55d4fe1b22eedacf81147213cfc7dee3d5b97a660d33f9a28"
Sep 15 12:05:23.839544 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:23.839503    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"]
Sep 15 12:05:23.840111 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:23.839773    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r" podUID="49b32be3-73ad-4d16-9fb2-f85cd743797d" containerName="payload-processing" containerID="cri-o://0cf7c5f1d3ab49de925aca99ef984f1bed5c209425b8e84ebb6aee3f25c9dc5c" gracePeriod=30
Sep 15 12:05:23.846722 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:23.846682    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"]
Sep 15 12:05:23.847154 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:23.847099    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt" podUID="44b6566d-420f-416e-bdfe-84ac73a0e6d2" containerName="payload-pre-processing" containerID="cri-o://ec94d1c4e5ef130313efc74751500765d2550eed46f666a3ef635d0ef5ade3a6" gracePeriod=30
Sep 15 12:05:24.036730 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.036695    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w"]
Sep 15 12:05:24.039471 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.039444    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25d1cb-v6b6w"]
Sep 15 12:05:24.104276 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.104219    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"
Sep 15 12:05:24.135333 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.135312    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"
Sep 15 12:05:24.151831 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.151810    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kzvt\" (UniqueName: \"kubernetes.io/projected/49b32be3-73ad-4d16-9fb2-f85cd743797d-kube-api-access-9kzvt\") pod \"49b32be3-73ad-4d16-9fb2-f85cd743797d\" (UID: \"49b32be3-73ad-4d16-9fb2-f85cd743797d\") "
Sep 15 12:05:24.151831 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.151844    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b32be3-73ad-4d16-9fb2-f85cd743797d-plugins-config-volume\") pod \"49b32be3-73ad-4d16-9fb2-f85cd743797d\" (UID: \"49b32be3-73ad-4d16-9fb2-f85cd743797d\") "
Sep 15 12:05:24.152072 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.151903    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h7j9\" (UniqueName: \"kubernetes.io/projected/44b6566d-420f-416e-bdfe-84ac73a0e6d2-kube-api-access-8h7j9\") pod \"44b6566d-420f-416e-bdfe-84ac73a0e6d2\" (UID: \"44b6566d-420f-416e-bdfe-84ac73a0e6d2\") "
Sep 15 12:05:24.152072 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.151941    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b6566d-420f-416e-bdfe-84ac73a0e6d2-plugins-config-volume\") pod \"44b6566d-420f-416e-bdfe-84ac73a0e6d2\" (UID: \"44b6566d-420f-416e-bdfe-84ac73a0e6d2\") "
Sep 15 12:05:24.152742 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.152707    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b6566d-420f-416e-bdfe-84ac73a0e6d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44b6566d-420f-416e-bdfe-84ac73a0e6d2" (UID: "44b6566d-420f-416e-bdfe-84ac73a0e6d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:05:24.153202 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.153175    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b32be3-73ad-4d16-9fb2-f85cd743797d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49b32be3-73ad-4d16-9fb2-f85cd743797d" (UID: "49b32be3-73ad-4d16-9fb2-f85cd743797d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:05:24.154645 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.154622    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b32be3-73ad-4d16-9fb2-f85cd743797d-kube-api-access-9kzvt" (OuterVolumeSpecName: "kube-api-access-9kzvt") pod "49b32be3-73ad-4d16-9fb2-f85cd743797d" (UID: "49b32be3-73ad-4d16-9fb2-f85cd743797d"). InnerVolumeSpecName "kube-api-access-9kzvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:05:24.155007 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.154978    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b6566d-420f-416e-bdfe-84ac73a0e6d2-kube-api-access-8h7j9" (OuterVolumeSpecName: "kube-api-access-8h7j9") pod "44b6566d-420f-416e-bdfe-84ac73a0e6d2" (UID: "44b6566d-420f-416e-bdfe-84ac73a0e6d2"). InnerVolumeSpecName "kube-api-access-8h7j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:05:24.253484 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.253449    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kzvt\" (UniqueName: \"kubernetes.io/projected/49b32be3-73ad-4d16-9fb2-f85cd743797d-kube-api-access-9kzvt\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:05:24.253484 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.253477    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b32be3-73ad-4d16-9fb2-f85cd743797d-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:05:24.253484 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.253490    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h7j9\" (UniqueName: \"kubernetes.io/projected/44b6566d-420f-416e-bdfe-84ac73a0e6d2-kube-api-access-8h7j9\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:05:24.253729 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.253501    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b6566d-420f-416e-bdfe-84ac73a0e6d2-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:05:24.556979 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.556945    2580 generic.go:356] "Generic (PLEG): container finished" podID="49b32be3-73ad-4d16-9fb2-f85cd743797d" containerID="0cf7c5f1d3ab49de925aca99ef984f1bed5c209425b8e84ebb6aee3f25c9dc5c" exitCode=0
Sep 15 12:05:24.557185 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.557002    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"
Sep 15 12:05:24.557185 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.557029    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r" event={"ID":"49b32be3-73ad-4d16-9fb2-f85cd743797d","Type":"ContainerDied","Data":"0cf7c5f1d3ab49de925aca99ef984f1bed5c209425b8e84ebb6aee3f25c9dc5c"}
Sep 15 12:05:24.557185 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.557083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r" event={"ID":"49b32be3-73ad-4d16-9fb2-f85cd743797d","Type":"ContainerDied","Data":"1a2f52f2b22e97932e884510976ac05f6e52f46a21262f23967257c1b065d8a5"}
Sep 15 12:05:24.557185 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.557114    2580 scope.go:117] "RemoveContainer" containerID="0cf7c5f1d3ab49de925aca99ef984f1bed5c209425b8e84ebb6aee3f25c9dc5c"
Sep 15 12:05:24.558377 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.558358    2580 generic.go:356] "Generic (PLEG): container finished" podID="44b6566d-420f-416e-bdfe-84ac73a0e6d2" containerID="ec94d1c4e5ef130313efc74751500765d2550eed46f666a3ef635d0ef5ade3a6" exitCode=0
Sep 15 12:05:24.558480 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.558398    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"
Sep 15 12:05:24.558480 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.558445    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt" event={"ID":"44b6566d-420f-416e-bdfe-84ac73a0e6d2","Type":"ContainerDied","Data":"ec94d1c4e5ef130313efc74751500765d2550eed46f666a3ef635d0ef5ade3a6"}
Sep 15 12:05:24.558590 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.558496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt" event={"ID":"44b6566d-420f-416e-bdfe-84ac73a0e6d2","Type":"ContainerDied","Data":"b1b66771a335304db7c5b6ec13287b93eafc2c93314642d42eeea06c0fcf1126"}
Sep 15 12:05:24.571091 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.571076    2580 scope.go:117] "RemoveContainer" containerID="0cf7c5f1d3ab49de925aca99ef984f1bed5c209425b8e84ebb6aee3f25c9dc5c"
Sep 15 12:05:24.571459 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:05:24.571434    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf7c5f1d3ab49de925aca99ef984f1bed5c209425b8e84ebb6aee3f25c9dc5c\": container with ID starting with 0cf7c5f1d3ab49de925aca99ef984f1bed5c209425b8e84ebb6aee3f25c9dc5c not found: ID does not exist" containerID="0cf7c5f1d3ab49de925aca99ef984f1bed5c209425b8e84ebb6aee3f25c9dc5c"
Sep 15 12:05:24.571543 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.571471    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf7c5f1d3ab49de925aca99ef984f1bed5c209425b8e84ebb6aee3f25c9dc5c"} err="failed to get container status \"0cf7c5f1d3ab49de925aca99ef984f1bed5c209425b8e84ebb6aee3f25c9dc5c\": rpc error: code = NotFound desc = could not find container \"0cf7c5f1d3ab49de925aca99ef984f1bed5c209425b8e84ebb6aee3f25c9dc5c\": container with ID starting with 0cf7c5f1d3ab49de925aca99ef984f1bed5c209425b8e84ebb6aee3f25c9dc5c not found: ID does not exist"
Sep 15 12:05:24.571543 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.571496    2580 scope.go:117] "RemoveContainer" containerID="ec94d1c4e5ef130313efc74751500765d2550eed46f666a3ef635d0ef5ade3a6"
Sep 15 12:05:24.580181 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.580164    2580 scope.go:117] "RemoveContainer" containerID="ec94d1c4e5ef130313efc74751500765d2550eed46f666a3ef635d0ef5ade3a6"
Sep 15 12:05:24.580455 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:05:24.580438    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec94d1c4e5ef130313efc74751500765d2550eed46f666a3ef635d0ef5ade3a6\": container with ID starting with ec94d1c4e5ef130313efc74751500765d2550eed46f666a3ef635d0ef5ade3a6 not found: ID does not exist" containerID="ec94d1c4e5ef130313efc74751500765d2550eed46f666a3ef635d0ef5ade3a6"
Sep 15 12:05:24.580509 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.580461    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec94d1c4e5ef130313efc74751500765d2550eed46f666a3ef635d0ef5ade3a6"} err="failed to get container status \"ec94d1c4e5ef130313efc74751500765d2550eed46f666a3ef635d0ef5ade3a6\": rpc error: code = NotFound desc = could not find container \"ec94d1c4e5ef130313efc74751500765d2550eed46f666a3ef635d0ef5ade3a6\": container with ID starting with ec94d1c4e5ef130313efc74751500765d2550eed46f666a3ef635d0ef5ade3a6 not found: ID does not exist"
Sep 15 12:05:24.587091 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.587071    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"]
Sep 15 12:05:24.591769 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.591748    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25d1cb-7fb89fc677-qgx2r"]
Sep 15 12:05:24.602854 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.602834    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"]
Sep 15 12:05:24.607526 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:24.607506    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25d1cb-884576fd-994kt"]
Sep 15 12:05:25.731831 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:25.731799    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44b6566d-420f-416e-bdfe-84ac73a0e6d2" path="/var/lib/kubelet/pods/44b6566d-420f-416e-bdfe-84ac73a0e6d2/volumes"
Sep 15 12:05:25.732216 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:25.732207    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49b32be3-73ad-4d16-9fb2-f85cd743797d" path="/var/lib/kubelet/pods/49b32be3-73ad-4d16-9fb2-f85cd743797d/volumes"
Sep 15 12:05:25.732504 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:25.732491    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9eb100d-e6e8-48df-8957-44704b70f0ae" path="/var/lib/kubelet/pods/e9eb100d-e6e8-48df-8957-44704b70f0ae/volumes"
Sep 15 12:05:31.739092 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.739055    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx"]
Sep 15 12:05:31.739546 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.739526    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b6566d-420f-416e-bdfe-84ac73a0e6d2" containerName="payload-pre-processing"
Sep 15 12:05:31.739546 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.739543    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b6566d-420f-416e-bdfe-84ac73a0e6d2" containerName="payload-pre-processing"
Sep 15 12:05:31.739669 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.739559    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b32be3-73ad-4d16-9fb2-f85cd743797d" containerName="payload-processing"
Sep 15 12:05:31.739669 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.739568    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b32be3-73ad-4d16-9fb2-f85cd743797d" containerName="payload-processing"
Sep 15 12:05:31.739669 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.739591    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9eb100d-e6e8-48df-8957-44704b70f0ae" containerName="revoke-keys"
Sep 15 12:05:31.739669 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.739598    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9eb100d-e6e8-48df-8957-44704b70f0ae" containerName="revoke-keys"
Sep 15 12:05:31.739837 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.739699    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9eb100d-e6e8-48df-8957-44704b70f0ae" containerName="revoke-keys"
Sep 15 12:05:31.739837 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.739712    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b32be3-73ad-4d16-9fb2-f85cd743797d" containerName="payload-processing"
Sep 15 12:05:31.739837 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.739723    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b6566d-420f-416e-bdfe-84ac73a0e6d2" containerName="payload-pre-processing"
Sep 15 12:05:31.744279 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.744253    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx"]
Sep 15 12:05:31.744404 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.744358    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx"
Sep 15 12:05:31.746852 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.746832    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:05:31.746967 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.746837    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:05:31.825400 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.825368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4504a4d9-f0de-4341-a7a0-776f159d1ebc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx\" (UID: \"4504a4d9-f0de-4341-a7a0-776f159d1ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx"
Sep 15 12:05:31.926538 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.926500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4504a4d9-f0de-4341-a7a0-776f159d1ebc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx\" (UID: \"4504a4d9-f0de-4341-a7a0-776f159d1ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx"
Sep 15 12:05:31.927183 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:31.927165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4504a4d9-f0de-4341-a7a0-776f159d1ebc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx\" (UID: \"4504a4d9-f0de-4341-a7a0-776f159d1ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx"
Sep 15 12:05:32.055224 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:32.055193    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx"
Sep 15 12:05:32.331331 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:32.331295    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx"]
Sep 15 12:05:32.332520 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:05:32.332494    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4504a4d9_f0de_4341_a7a0_776f159d1ebc.slice/crio-910f79e7276290e56abde64a85c789f9fb94a2945b183f7d6d23d102e04195a1 WatchSource:0}: Error finding container 910f79e7276290e56abde64a85c789f9fb94a2945b183f7d6d23d102e04195a1: Status 404 returned error can't find the container with id 910f79e7276290e56abde64a85c789f9fb94a2945b183f7d6d23d102e04195a1
Sep 15 12:05:32.614261 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:32.614224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx" event={"ID":"4504a4d9-f0de-4341-a7a0-776f159d1ebc","Type":"ContainerStarted","Data":"dcc823f2b44a5e0b2eefda6627c2dbdfa64e5d20ba94ec6bdb2b4bfdfa54b463"}
Sep 15 12:05:32.614408 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:32.614287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx" event={"ID":"4504a4d9-f0de-4341-a7a0-776f159d1ebc","Type":"ContainerStarted","Data":"910f79e7276290e56abde64a85c789f9fb94a2945b183f7d6d23d102e04195a1"}
Sep 15 12:05:32.633194 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:32.632971    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx" podStartSLOduration=1.63295251 podStartE2EDuration="1.63295251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:05:32.632645806 +0000 UTC m=+1621.413217193" watchObservedRunningTime="2026-09-15 12:05:32.63295251 +0000 UTC m=+1621.413523898"
Sep 15 12:05:33.622049 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:33.622011    2580 generic.go:356] "Generic (PLEG): container finished" podID="4504a4d9-f0de-4341-a7a0-776f159d1ebc" containerID="dcc823f2b44a5e0b2eefda6627c2dbdfa64e5d20ba94ec6bdb2b4bfdfa54b463" exitCode=0
Sep 15 12:05:33.622446 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:33.622060    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx" event={"ID":"4504a4d9-f0de-4341-a7a0-776f159d1ebc","Type":"ContainerDied","Data":"dcc823f2b44a5e0b2eefda6627c2dbdfa64e5d20ba94ec6bdb2b4bfdfa54b463"}
Sep 15 12:05:34.763781 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:34.763754    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx"
Sep 15 12:05:34.856344 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:34.856304    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4504a4d9-f0de-4341-a7a0-776f159d1ebc-maas-api-service-ca\") pod \"4504a4d9-f0de-4341-a7a0-776f159d1ebc\" (UID: \"4504a4d9-f0de-4341-a7a0-776f159d1ebc\") "
Sep 15 12:05:34.856639 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:34.856618    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4504a4d9-f0de-4341-a7a0-776f159d1ebc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4504a4d9-f0de-4341-a7a0-776f159d1ebc" (UID: "4504a4d9-f0de-4341-a7a0-776f159d1ebc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:05:34.957374 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:34.957342    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4504a4d9-f0de-4341-a7a0-776f159d1ebc-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:05:35.635848 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:35.635822    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx"
Sep 15 12:05:35.636065 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:35.635866    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx" event={"ID":"4504a4d9-f0de-4341-a7a0-776f159d1ebc","Type":"ContainerDied","Data":"910f79e7276290e56abde64a85c789f9fb94a2945b183f7d6d23d102e04195a1"}
Sep 15 12:05:35.636065 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:35.635939    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="910f79e7276290e56abde64a85c789f9fb94a2945b183f7d6d23d102e04195a1"
Sep 15 12:05:41.465374 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.465340    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"]
Sep 15 12:05:41.465742 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.465729    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4504a4d9-f0de-4341-a7a0-776f159d1ebc" containerName="revoke-keys"
Sep 15 12:05:41.465742 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.465743    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4504a4d9-f0de-4341-a7a0-776f159d1ebc" containerName="revoke-keys"
Sep 15 12:05:41.465836 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.465815    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4504a4d9-f0de-4341-a7a0-776f159d1ebc" containerName="revoke-keys"
Sep 15 12:05:41.468821 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.468805    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.471155 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.471134    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-29d630\"/\"embedding-w0-29d630-kserve-self-signed-certs\""
Sep 15 12:05:41.477362 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.477338    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"]
Sep 15 12:05:41.521570 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.521543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-dshm\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.521724 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.521660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-tmp-dir\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.521724 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.521692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zblrb\" (UniqueName: \"kubernetes.io/projected/5bbe8724-a854-4772-9dc7-f70be0953337-kube-api-access-zblrb\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.521724 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.521716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-home\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.521916 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.521734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-model-cache\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.521916 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.521832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bbe8724-a854-4772-9dc7-f70be0953337-tls-certs\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.623107 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.623076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bbe8724-a854-4772-9dc7-f70be0953337-tls-certs\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.623269 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.623126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-dshm\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.623269 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.623188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-tmp-dir\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.623269 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.623211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zblrb\" (UniqueName: \"kubernetes.io/projected/5bbe8724-a854-4772-9dc7-f70be0953337-kube-api-access-zblrb\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.623269 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.623230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-home\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.623269 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.623255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-model-cache\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.623623 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.623596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-tmp-dir\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.623698 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.623644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-model-cache\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.623698 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.623673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-home\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.625563 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.625541    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-dshm\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.625709 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.625692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bbe8724-a854-4772-9dc7-f70be0953337-tls-certs\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.632393 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.632341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zblrb\" (UniqueName: \"kubernetes.io/projected/5bbe8724-a854-4772-9dc7-f70be0953337-kube-api-access-zblrb\") pod \"embedding-w0-29d630-kserve-5569b4d6d-96hmx\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") " pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:41.779306 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:41.779225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:42.056434 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:42.056407    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"]
Sep 15 12:05:42.057799 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:05:42.057771    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bbe8724_a854_4772_9dc7_f70be0953337.slice/crio-452ba76f0ba7232d604a971db9937d22bf8d0a48a1c6ae2901847073e0fd95f8 WatchSource:0}: Error finding container 452ba76f0ba7232d604a971db9937d22bf8d0a48a1c6ae2901847073e0fd95f8: Status 404 returned error can't find the container with id 452ba76f0ba7232d604a971db9937d22bf8d0a48a1c6ae2901847073e0fd95f8
Sep 15 12:05:42.681609 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:42.681570    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx" event={"ID":"5bbe8724-a854-4772-9dc7-f70be0953337","Type":"ContainerStarted","Data":"cec3297412c564acf8a0997bf720f99f1473493df31f5a8c44dced7e6d30a7cc"}
Sep 15 12:05:42.682028 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:42.681622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx" event={"ID":"5bbe8724-a854-4772-9dc7-f70be0953337","Type":"ContainerStarted","Data":"452ba76f0ba7232d604a971db9937d22bf8d0a48a1c6ae2901847073e0fd95f8"}
Sep 15 12:05:42.700808 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:42.700755    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx" podStartSLOduration=1.700738576 podStartE2EDuration="1.700738576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:05:42.698563302 +0000 UTC m=+1631.479134690" watchObservedRunningTime="2026-09-15 12:05:42.700738576 +0000 UTC m=+1631.481309963"
Sep 15 12:05:47.446112 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.446077    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"]
Sep 15 12:05:47.446833 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.446794    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f" podUID="bfde6835-a1ea-4ea9-aad6-42aae307cfa7" containerName="payload-processing" containerID="cri-o://29ddb587594dc68799f1f413726494b9d9e4fd498cac0d03f088b025c941f01f" gracePeriod=30
Sep 15 12:05:47.636698 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.636668    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx"]
Sep 15 12:05:47.641251 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.641219    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0848f9-c2hmx"]
Sep 15 12:05:47.699964 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.699906    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"
Sep 15 12:05:47.712578 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.712555    2580 generic.go:356] "Generic (PLEG): container finished" podID="bfde6835-a1ea-4ea9-aad6-42aae307cfa7" containerID="29ddb587594dc68799f1f413726494b9d9e4fd498cac0d03f088b025c941f01f" exitCode=0
Sep 15 12:05:47.712688 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.712594    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"
Sep 15 12:05:47.712688 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.712636    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f" event={"ID":"bfde6835-a1ea-4ea9-aad6-42aae307cfa7","Type":"ContainerDied","Data":"29ddb587594dc68799f1f413726494b9d9e4fd498cac0d03f088b025c941f01f"}
Sep 15 12:05:47.712756 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.712709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f" event={"ID":"bfde6835-a1ea-4ea9-aad6-42aae307cfa7","Type":"ContainerDied","Data":"dd143e114194199e214d7ee3f399516e6982f8134b2283b9a3d02b6d7df73d3d"}
Sep 15 12:05:47.712756 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.712738    2580 scope.go:117] "RemoveContainer" containerID="29ddb587594dc68799f1f413726494b9d9e4fd498cac0d03f088b025c941f01f"
Sep 15 12:05:47.723820 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.723799    2580 scope.go:117] "RemoveContainer" containerID="29ddb587594dc68799f1f413726494b9d9e4fd498cac0d03f088b025c941f01f"
Sep 15 12:05:47.724278 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:05:47.724254    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ddb587594dc68799f1f413726494b9d9e4fd498cac0d03f088b025c941f01f\": container with ID starting with 29ddb587594dc68799f1f413726494b9d9e4fd498cac0d03f088b025c941f01f not found: ID does not exist" containerID="29ddb587594dc68799f1f413726494b9d9e4fd498cac0d03f088b025c941f01f"
Sep 15 12:05:47.724487 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.724448    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ddb587594dc68799f1f413726494b9d9e4fd498cac0d03f088b025c941f01f"} err="failed to get container status \"29ddb587594dc68799f1f413726494b9d9e4fd498cac0d03f088b025c941f01f\": rpc error: code = NotFound desc = could not find container \"29ddb587594dc68799f1f413726494b9d9e4fd498cac0d03f088b025c941f01f\": container with ID starting with 29ddb587594dc68799f1f413726494b9d9e4fd498cac0d03f088b025c941f01f not found: ID does not exist"
Sep 15 12:05:47.731046 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.731022    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4504a4d9-f0de-4341-a7a0-776f159d1ebc" path="/var/lib/kubelet/pods/4504a4d9-f0de-4341-a7a0-776f159d1ebc/volumes"
Sep 15 12:05:47.784292 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.784268    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4q26\" (UniqueName: \"kubernetes.io/projected/bfde6835-a1ea-4ea9-aad6-42aae307cfa7-kube-api-access-s4q26\") pod \"bfde6835-a1ea-4ea9-aad6-42aae307cfa7\" (UID: \"bfde6835-a1ea-4ea9-aad6-42aae307cfa7\") "
Sep 15 12:05:47.784387 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.784350    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfde6835-a1ea-4ea9-aad6-42aae307cfa7-plugins-config-volume\") pod \"bfde6835-a1ea-4ea9-aad6-42aae307cfa7\" (UID: \"bfde6835-a1ea-4ea9-aad6-42aae307cfa7\") "
Sep 15 12:05:47.784734 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.784715    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfde6835-a1ea-4ea9-aad6-42aae307cfa7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bfde6835-a1ea-4ea9-aad6-42aae307cfa7" (UID: "bfde6835-a1ea-4ea9-aad6-42aae307cfa7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:05:47.786349 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.786323    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfde6835-a1ea-4ea9-aad6-42aae307cfa7-kube-api-access-s4q26" (OuterVolumeSpecName: "kube-api-access-s4q26") pod "bfde6835-a1ea-4ea9-aad6-42aae307cfa7" (UID: "bfde6835-a1ea-4ea9-aad6-42aae307cfa7"). InnerVolumeSpecName "kube-api-access-s4q26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:05:47.886123 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.886089    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4q26\" (UniqueName: \"kubernetes.io/projected/bfde6835-a1ea-4ea9-aad6-42aae307cfa7-kube-api-access-s4q26\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:05:47.886123 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:47.886119    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfde6835-a1ea-4ea9-aad6-42aae307cfa7-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:05:48.041067 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:48.041034    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"]
Sep 15 12:05:48.044304 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:48.044279    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0848f9-d89f976c5-pk87f"]
Sep 15 12:05:49.730890 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:49.730843    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfde6835-a1ea-4ea9-aad6-42aae307cfa7" path="/var/lib/kubelet/pods/bfde6835-a1ea-4ea9-aad6-42aae307cfa7/volumes"
Sep 15 12:05:51.779760 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:51.779731    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:51.780180 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:51.779774    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:51.792398 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:51.792371    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:52.755051 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:52.755016    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:05:55.741218 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:55.741184    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd"]
Sep 15 12:05:55.741693 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:55.741577    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfde6835-a1ea-4ea9-aad6-42aae307cfa7" containerName="payload-processing"
Sep 15 12:05:55.741693 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:55.741591    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfde6835-a1ea-4ea9-aad6-42aae307cfa7" containerName="payload-processing"
Sep 15 12:05:55.741693 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:55.741673    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfde6835-a1ea-4ea9-aad6-42aae307cfa7" containerName="payload-processing"
Sep 15 12:05:55.746054 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:55.746036    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd"
Sep 15 12:05:55.748688 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:55.748658    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:05:55.748688 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:55.748671    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:05:55.752096 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:55.752068    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd"]
Sep 15 12:05:55.865979 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:55.865945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e977a5f2-5214-4262-9140-3b899e89d83b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd\" (UID: \"e977a5f2-5214-4262-9140-3b899e89d83b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd"
Sep 15 12:05:55.967281 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:55.967242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e977a5f2-5214-4262-9140-3b899e89d83b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd\" (UID: \"e977a5f2-5214-4262-9140-3b899e89d83b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd"
Sep 15 12:05:55.967927 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:55.967909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e977a5f2-5214-4262-9140-3b899e89d83b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd\" (UID: \"e977a5f2-5214-4262-9140-3b899e89d83b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd"
Sep 15 12:05:56.059310 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:56.059229    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd"
Sep 15 12:05:56.336529 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:56.336496    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd"]
Sep 15 12:05:56.337617 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:05:56.337588    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode977a5f2_5214_4262_9140_3b899e89d83b.slice/crio-a5a618ad5d9ad53bfad362bdd9a45459e2afaca04a44539a420b89e14e2296b5 WatchSource:0}: Error finding container a5a618ad5d9ad53bfad362bdd9a45459e2afaca04a44539a420b89e14e2296b5: Status 404 returned error can't find the container with id a5a618ad5d9ad53bfad362bdd9a45459e2afaca04a44539a420b89e14e2296b5
Sep 15 12:05:56.768764 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:56.768722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd" event={"ID":"e977a5f2-5214-4262-9140-3b899e89d83b","Type":"ContainerStarted","Data":"da2f890e80d371a8df7dc0823e0015cee6a1358c0375ef7a397b818ef50e4176"}
Sep 15 12:05:56.769130 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:56.768786    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd" event={"ID":"e977a5f2-5214-4262-9140-3b899e89d83b","Type":"ContainerStarted","Data":"a5a618ad5d9ad53bfad362bdd9a45459e2afaca04a44539a420b89e14e2296b5"}
Sep 15 12:05:56.787189 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:56.787139    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd" podStartSLOduration=1.7871243589999999 podStartE2EDuration="1.787124359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:05:56.784604278 +0000 UTC m=+1645.565175657" watchObservedRunningTime="2026-09-15 12:05:56.787124359 +0000 UTC m=+1645.567695746"
Sep 15 12:05:57.776620 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:57.776584    2580 generic.go:356] "Generic (PLEG): container finished" podID="e977a5f2-5214-4262-9140-3b899e89d83b" containerID="da2f890e80d371a8df7dc0823e0015cee6a1358c0375ef7a397b818ef50e4176" exitCode=0
Sep 15 12:05:57.777039 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:57.776671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd" event={"ID":"e977a5f2-5214-4262-9140-3b899e89d83b","Type":"ContainerDied","Data":"da2f890e80d371a8df7dc0823e0015cee6a1358c0375ef7a397b818ef50e4176"}
Sep 15 12:05:58.911973 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:58.911952    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd"
Sep 15 12:05:59.099277 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:59.099191    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e977a5f2-5214-4262-9140-3b899e89d83b-maas-api-service-ca\") pod \"e977a5f2-5214-4262-9140-3b899e89d83b\" (UID: \"e977a5f2-5214-4262-9140-3b899e89d83b\") "
Sep 15 12:05:59.099604 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:59.099569    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e977a5f2-5214-4262-9140-3b899e89d83b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e977a5f2-5214-4262-9140-3b899e89d83b" (UID: "e977a5f2-5214-4262-9140-3b899e89d83b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:05:59.200743 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:59.200708    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e977a5f2-5214-4262-9140-3b899e89d83b-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:05:59.790717 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:59.790682    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd"
Sep 15 12:05:59.790717 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:59.790695    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd" event={"ID":"e977a5f2-5214-4262-9140-3b899e89d83b","Type":"ContainerDied","Data":"a5a618ad5d9ad53bfad362bdd9a45459e2afaca04a44539a420b89e14e2296b5"}
Sep 15 12:05:59.790932 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:05:59.790751    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5a618ad5d9ad53bfad362bdd9a45459e2afaca04a44539a420b89e14e2296b5"
Sep 15 12:06:06.277383 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.277350    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ec8230-29824560-rnwkq"]
Sep 15 12:06:06.280596 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.280575    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ec8230-29824560-rnwkq"]
Sep 15 12:06:06.316311 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.316281    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"]
Sep 15 12:06:06.316585 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.316544    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr" podUID="e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c" containerName="payload-pre-processing" containerID="cri-o://f9448f0d3c1820d7f68d20d6b8d8e57f0892cc935a7877023e2f9acec51cc616" gracePeriod=30
Sep 15 12:06:06.512717 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.512687    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd"]
Sep 15 12:06:06.515436 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.515407    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec8230-drdqd"]
Sep 15 12:06:06.573966 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.573942    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"
Sep 15 12:06:06.668105 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.668068    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c-plugins-config-volume\") pod \"e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c\" (UID: \"e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c\") "
Sep 15 12:06:06.668356 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.668163    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57xtm\" (UniqueName: \"kubernetes.io/projected/e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c-kube-api-access-57xtm\") pod \"e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c\" (UID: \"e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c\") "
Sep 15 12:06:06.668539 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.668511    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c" (UID: "e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:06:06.670421 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.670394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c-kube-api-access-57xtm" (OuterVolumeSpecName: "kube-api-access-57xtm") pod "e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c" (UID: "e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c"). InnerVolumeSpecName "kube-api-access-57xtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:06:06.769666 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.769617    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:06:06.769666 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.769657    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57xtm\" (UniqueName: \"kubernetes.io/projected/e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c-kube-api-access-57xtm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:06:06.834724 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.834627    2580 generic.go:356] "Generic (PLEG): container finished" podID="e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c" containerID="f9448f0d3c1820d7f68d20d6b8d8e57f0892cc935a7877023e2f9acec51cc616" exitCode=0
Sep 15 12:06:06.834724 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.834674    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"
Sep 15 12:06:06.834979 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.834718    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr" event={"ID":"e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c","Type":"ContainerDied","Data":"f9448f0d3c1820d7f68d20d6b8d8e57f0892cc935a7877023e2f9acec51cc616"}
Sep 15 12:06:06.834979 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.834775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr" event={"ID":"e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c","Type":"ContainerDied","Data":"b3c334597db7f6baec1a471436722ee9dfe6182c42452772351c1e7efd90c8c1"}
Sep 15 12:06:06.834979 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.834798    2580 scope.go:117] "RemoveContainer" containerID="f9448f0d3c1820d7f68d20d6b8d8e57f0892cc935a7877023e2f9acec51cc616"
Sep 15 12:06:06.844449 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.844428    2580 scope.go:117] "RemoveContainer" containerID="f9448f0d3c1820d7f68d20d6b8d8e57f0892cc935a7877023e2f9acec51cc616"
Sep 15 12:06:06.844714 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:06:06.844693    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9448f0d3c1820d7f68d20d6b8d8e57f0892cc935a7877023e2f9acec51cc616\": container with ID starting with f9448f0d3c1820d7f68d20d6b8d8e57f0892cc935a7877023e2f9acec51cc616 not found: ID does not exist" containerID="f9448f0d3c1820d7f68d20d6b8d8e57f0892cc935a7877023e2f9acec51cc616"
Sep 15 12:06:06.844785 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.844726    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9448f0d3c1820d7f68d20d6b8d8e57f0892cc935a7877023e2f9acec51cc616"} err="failed to get container status \"f9448f0d3c1820d7f68d20d6b8d8e57f0892cc935a7877023e2f9acec51cc616\": rpc error: code = NotFound desc = could not find container \"f9448f0d3c1820d7f68d20d6b8d8e57f0892cc935a7877023e2f9acec51cc616\": container with ID starting with f9448f0d3c1820d7f68d20d6b8d8e57f0892cc935a7877023e2f9acec51cc616 not found: ID does not exist"
Sep 15 12:06:06.859599 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.859571    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"]
Sep 15 12:06:06.867999 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:06.864778    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec8230-7657cc6b4b-lntqr"]
Sep 15 12:06:07.731460 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:07.731420    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d52dc0ba-2fc3-44ba-b964-071011b0e912" path="/var/lib/kubelet/pods/d52dc0ba-2fc3-44ba-b964-071011b0e912/volumes"
Sep 15 12:06:07.732002 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:07.731981    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c" path="/var/lib/kubelet/pods/e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c/volumes"
Sep 15 12:06:07.732380 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:07.732361    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e977a5f2-5214-4262-9140-3b899e89d83b" path="/var/lib/kubelet/pods/e977a5f2-5214-4262-9140-3b899e89d83b/volumes"
Sep 15 12:06:13.844342 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:13.844301    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72"]
Sep 15 12:06:13.844840 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:13.844819    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e977a5f2-5214-4262-9140-3b899e89d83b" containerName="revoke-keys"
Sep 15 12:06:13.844960 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:13.844844    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e977a5f2-5214-4262-9140-3b899e89d83b" containerName="revoke-keys"
Sep 15 12:06:13.844960 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:13.844908    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c" containerName="payload-pre-processing"
Sep 15 12:06:13.844960 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:13.844919    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c" containerName="payload-pre-processing"
Sep 15 12:06:13.845113 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:13.845034    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e10f1f44-4abe-4b7e-949d-f6e1b4cfdc9c" containerName="payload-pre-processing"
Sep 15 12:06:13.845113 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:13.845048    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e977a5f2-5214-4262-9140-3b899e89d83b" containerName="revoke-keys"
Sep 15 12:06:13.849478 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:13.849455    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72"
Sep 15 12:06:13.852148 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:13.852121    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:06:13.852267 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:13.852119    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:06:13.854769 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:13.854747    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72"]
Sep 15 12:06:13.942995 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:13.942963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4fd1cb5-c3cf-4223-9972-a39a2e41c707-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72\" (UID: \"e4fd1cb5-c3cf-4223-9972-a39a2e41c707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72"
Sep 15 12:06:14.044194 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:14.044154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4fd1cb5-c3cf-4223-9972-a39a2e41c707-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72\" (UID: \"e4fd1cb5-c3cf-4223-9972-a39a2e41c707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72"
Sep 15 12:06:14.044781 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:14.044762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4fd1cb5-c3cf-4223-9972-a39a2e41c707-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72\" (UID: \"e4fd1cb5-c3cf-4223-9972-a39a2e41c707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72"
Sep 15 12:06:14.161725 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:14.161639    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72"
Sep 15 12:06:14.438257 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:14.438231    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72"]
Sep 15 12:06:14.440027 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:06:14.439996    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fd1cb5_c3cf_4223_9972_a39a2e41c707.slice/crio-6e56cba0779c9d8442c63bcef9c941050dda23c194ed9464c6d48d1639dab923 WatchSource:0}: Error finding container 6e56cba0779c9d8442c63bcef9c941050dda23c194ed9464c6d48d1639dab923: Status 404 returned error can't find the container with id 6e56cba0779c9d8442c63bcef9c941050dda23c194ed9464c6d48d1639dab923
Sep 15 12:06:14.890171 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:14.890134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72" event={"ID":"e4fd1cb5-c3cf-4223-9972-a39a2e41c707","Type":"ContainerStarted","Data":"28fe1ef5b1dc2fbd8ddcf31905fd9b6c5da4984b00b217f5287de0ed804eef2c"}
Sep 15 12:06:14.890553 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:14.890196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72" event={"ID":"e4fd1cb5-c3cf-4223-9972-a39a2e41c707","Type":"ContainerStarted","Data":"6e56cba0779c9d8442c63bcef9c941050dda23c194ed9464c6d48d1639dab923"}
Sep 15 12:06:14.914044 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:14.913992    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72" podStartSLOduration=1.91397717 podStartE2EDuration="1.91397717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:06:14.911993287 +0000 UTC m=+1663.692564674" watchObservedRunningTime="2026-09-15 12:06:14.91397717 +0000 UTC m=+1663.694548557"
Sep 15 12:06:15.898995 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:15.898962    2580 generic.go:356] "Generic (PLEG): container finished" podID="e4fd1cb5-c3cf-4223-9972-a39a2e41c707" containerID="28fe1ef5b1dc2fbd8ddcf31905fd9b6c5da4984b00b217f5287de0ed804eef2c" exitCode=0
Sep 15 12:06:15.899398 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:15.899045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72" event={"ID":"e4fd1cb5-c3cf-4223-9972-a39a2e41c707","Type":"ContainerDied","Data":"28fe1ef5b1dc2fbd8ddcf31905fd9b6c5da4984b00b217f5287de0ed804eef2c"}
Sep 15 12:06:17.035462 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:17.035436    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72"
Sep 15 12:06:17.174412 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:17.174382    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4fd1cb5-c3cf-4223-9972-a39a2e41c707-maas-api-service-ca\") pod \"e4fd1cb5-c3cf-4223-9972-a39a2e41c707\" (UID: \"e4fd1cb5-c3cf-4223-9972-a39a2e41c707\") "
Sep 15 12:06:17.174803 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:17.174774    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4fd1cb5-c3cf-4223-9972-a39a2e41c707-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e4fd1cb5-c3cf-4223-9972-a39a2e41c707" (UID: "e4fd1cb5-c3cf-4223-9972-a39a2e41c707"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:06:17.275206 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:17.275174    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4fd1cb5-c3cf-4223-9972-a39a2e41c707-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:06:17.912210 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:17.912178    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72"
Sep 15 12:06:17.912383 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:17.912222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72" event={"ID":"e4fd1cb5-c3cf-4223-9972-a39a2e41c707","Type":"ContainerDied","Data":"6e56cba0779c9d8442c63bcef9c941050dda23c194ed9464c6d48d1639dab923"}
Sep 15 12:06:17.912383 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:17.912274    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e56cba0779c9d8442c63bcef9c941050dda23c194ed9464c6d48d1639dab923"
Sep 15 12:06:24.382775 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.382739    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6dca39-29824560-wkqvn"]
Sep 15 12:06:24.392977 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.392947    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6dca39-29824560-wkqvn"]
Sep 15 12:06:24.450268 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.450236    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"]
Sep 15 12:06:24.450493 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.450451    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h" podUID="c6978c6b-5b97-46a5-b3b6-8b7d81e141f1" containerName="payload-pre-processing" containerID="cri-o://2078637080ab95bead9eda00437931a78b36501495931abdc22f1aa8e69bb240" gracePeriod=30
Sep 15 12:06:24.647908 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.647820    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72"]
Sep 15 12:06:24.652320 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.652290    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6dca39-c2l72"]
Sep 15 12:06:24.709483 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.709460    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"
Sep 15 12:06:24.848162 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.848125    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6978c6b-5b97-46a5-b3b6-8b7d81e141f1-plugins-config-volume\") pod \"c6978c6b-5b97-46a5-b3b6-8b7d81e141f1\" (UID: \"c6978c6b-5b97-46a5-b3b6-8b7d81e141f1\") "
Sep 15 12:06:24.848362 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.848214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz9vf\" (UniqueName: \"kubernetes.io/projected/c6978c6b-5b97-46a5-b3b6-8b7d81e141f1-kube-api-access-fz9vf\") pod \"c6978c6b-5b97-46a5-b3b6-8b7d81e141f1\" (UID: \"c6978c6b-5b97-46a5-b3b6-8b7d81e141f1\") "
Sep 15 12:06:24.848624 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.848566    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6978c6b-5b97-46a5-b3b6-8b7d81e141f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c6978c6b-5b97-46a5-b3b6-8b7d81e141f1" (UID: "c6978c6b-5b97-46a5-b3b6-8b7d81e141f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:06:24.850325 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.850297    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6978c6b-5b97-46a5-b3b6-8b7d81e141f1-kube-api-access-fz9vf" (OuterVolumeSpecName: "kube-api-access-fz9vf") pod "c6978c6b-5b97-46a5-b3b6-8b7d81e141f1" (UID: "c6978c6b-5b97-46a5-b3b6-8b7d81e141f1"). InnerVolumeSpecName "kube-api-access-fz9vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:06:24.949445 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.949414    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6978c6b-5b97-46a5-b3b6-8b7d81e141f1-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:06:24.949445 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.949444    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz9vf\" (UniqueName: \"kubernetes.io/projected/c6978c6b-5b97-46a5-b3b6-8b7d81e141f1-kube-api-access-fz9vf\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:06:24.956439 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.956405    2580 generic.go:356] "Generic (PLEG): container finished" podID="c6978c6b-5b97-46a5-b3b6-8b7d81e141f1" containerID="2078637080ab95bead9eda00437931a78b36501495931abdc22f1aa8e69bb240" exitCode=0
Sep 15 12:06:24.956567 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.956457    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"
Sep 15 12:06:24.956567 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.956492    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h" event={"ID":"c6978c6b-5b97-46a5-b3b6-8b7d81e141f1","Type":"ContainerDied","Data":"2078637080ab95bead9eda00437931a78b36501495931abdc22f1aa8e69bb240"}
Sep 15 12:06:24.956567 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.956560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h" event={"ID":"c6978c6b-5b97-46a5-b3b6-8b7d81e141f1","Type":"ContainerDied","Data":"ad77af8c656385e0cc68e4ad136ef130ce0b87d9b2230b29c54b2f370addf086"}
Sep 15 12:06:24.956743 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.956590    2580 scope.go:117] "RemoveContainer" containerID="2078637080ab95bead9eda00437931a78b36501495931abdc22f1aa8e69bb240"
Sep 15 12:06:24.967182 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.967161    2580 scope.go:117] "RemoveContainer" containerID="2078637080ab95bead9eda00437931a78b36501495931abdc22f1aa8e69bb240"
Sep 15 12:06:24.967449 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:06:24.967431    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2078637080ab95bead9eda00437931a78b36501495931abdc22f1aa8e69bb240\": container with ID starting with 2078637080ab95bead9eda00437931a78b36501495931abdc22f1aa8e69bb240 not found: ID does not exist" containerID="2078637080ab95bead9eda00437931a78b36501495931abdc22f1aa8e69bb240"
Sep 15 12:06:24.967515 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.967463    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2078637080ab95bead9eda00437931a78b36501495931abdc22f1aa8e69bb240"} err="failed to get container status \"2078637080ab95bead9eda00437931a78b36501495931abdc22f1aa8e69bb240\": rpc error: code = NotFound desc = could not find container \"2078637080ab95bead9eda00437931a78b36501495931abdc22f1aa8e69bb240\": container with ID starting with 2078637080ab95bead9eda00437931a78b36501495931abdc22f1aa8e69bb240 not found: ID does not exist"
Sep 15 12:06:24.988038 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.988013    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"]
Sep 15 12:06:24.999091 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:24.999067    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6dca39-89968d878-bn79h"]
Sep 15 12:06:25.731188 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:25.731152    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c54ccce0-837b-4b9b-867d-5c9d45e33719" path="/var/lib/kubelet/pods/c54ccce0-837b-4b9b-867d-5c9d45e33719/volumes"
Sep 15 12:06:25.731566 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:25.731504    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6978c6b-5b97-46a5-b3b6-8b7d81e141f1" path="/var/lib/kubelet/pods/c6978c6b-5b97-46a5-b3b6-8b7d81e141f1/volumes"
Sep 15 12:06:25.731807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:25.731794    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4fd1cb5-c3cf-4223-9972-a39a2e41c707" path="/var/lib/kubelet/pods/e4fd1cb5-c3cf-4223-9972-a39a2e41c707/volumes"
Sep 15 12:06:32.032712 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:32.032684    2580 scope.go:117] "RemoveContainer" containerID="a03080c439155165b774d3fcacd1d2f492220f451bf7a2a9f178b7f78576f382"
Sep 15 12:06:32.041344 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:32.041322    2580 scope.go:117] "RemoveContainer" containerID="47d612e48b3c7cbf404f78fd21d8e65cdd660fc75e9c09a18cae2eafc544f43b"
Sep 15 12:06:32.049333 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:32.049312    2580 scope.go:117] "RemoveContainer" containerID="e43b7f846dd9f4506ee5ca8798b6bc5d9c3671b4e4dcc779c52d405b04f2cbd9"
Sep 15 12:06:32.057056 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:32.057035    2580 scope.go:117] "RemoveContainer" containerID="c42149d494e9e48dc57341ffff2cb317dfc3ba53a6683917449aee06139f135b"
Sep 15 12:06:46.639518 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.639477    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6"]
Sep 15 12:06:46.639933 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.639829    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4fd1cb5-c3cf-4223-9972-a39a2e41c707" containerName="revoke-keys"
Sep 15 12:06:46.639933 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.639840    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4fd1cb5-c3cf-4223-9972-a39a2e41c707" containerName="revoke-keys"
Sep 15 12:06:46.639933 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.639848    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6978c6b-5b97-46a5-b3b6-8b7d81e141f1" containerName="payload-pre-processing"
Sep 15 12:06:46.639933 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.639855    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6978c6b-5b97-46a5-b3b6-8b7d81e141f1" containerName="payload-pre-processing"
Sep 15 12:06:46.640081 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.639944    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6978c6b-5b97-46a5-b3b6-8b7d81e141f1" containerName="payload-pre-processing"
Sep 15 12:06:46.640081 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.639952    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4fd1cb5-c3cf-4223-9972-a39a2e41c707" containerName="revoke-keys"
Sep 15 12:06:46.642101 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.642084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6"
Sep 15 12:06:46.644306 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.644283    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:06:46.644499 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.644483    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:06:46.650771 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.650746    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6"]
Sep 15 12:06:46.748704 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.748666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de48197c-e0dc-4aa1-9e86-51cfd16f8025-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6\" (UID: \"de48197c-e0dc-4aa1-9e86-51cfd16f8025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6"
Sep 15 12:06:46.849696 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.849663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de48197c-e0dc-4aa1-9e86-51cfd16f8025-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6\" (UID: \"de48197c-e0dc-4aa1-9e86-51cfd16f8025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6"
Sep 15 12:06:46.850270 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.850252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de48197c-e0dc-4aa1-9e86-51cfd16f8025-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6\" (UID: \"de48197c-e0dc-4aa1-9e86-51cfd16f8025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6"
Sep 15 12:06:46.953944 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:46.953914    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6"
Sep 15 12:06:47.228763 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:47.228737    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6"]
Sep 15 12:06:47.229793 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:06:47.229771    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde48197c_e0dc_4aa1_9e86_51cfd16f8025.slice/crio-38da8da15088c78489765a293517f280f94d416208d514b7441956e856a2e461 WatchSource:0}: Error finding container 38da8da15088c78489765a293517f280f94d416208d514b7441956e856a2e461: Status 404 returned error can't find the container with id 38da8da15088c78489765a293517f280f94d416208d514b7441956e856a2e461
Sep 15 12:06:48.106062 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:48.106025    2580 generic.go:356] "Generic (PLEG): container finished" podID="de48197c-e0dc-4aa1-9e86-51cfd16f8025" containerID="080ccf166b3d5956474a5165c88a4c254a20ceb887ac1484bf45de6b2d6479f5" exitCode=0
Sep 15 12:06:48.106481 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:48.106067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6" event={"ID":"de48197c-e0dc-4aa1-9e86-51cfd16f8025","Type":"ContainerDied","Data":"080ccf166b3d5956474a5165c88a4c254a20ceb887ac1484bf45de6b2d6479f5"}
Sep 15 12:06:48.106481 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:48.106123    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6" event={"ID":"de48197c-e0dc-4aa1-9e86-51cfd16f8025","Type":"ContainerStarted","Data":"38da8da15088c78489765a293517f280f94d416208d514b7441956e856a2e461"}
Sep 15 12:06:49.247149 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:49.247124    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6"
Sep 15 12:06:49.275888 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:49.275844    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de48197c-e0dc-4aa1-9e86-51cfd16f8025-maas-api-service-ca\") pod \"de48197c-e0dc-4aa1-9e86-51cfd16f8025\" (UID: \"de48197c-e0dc-4aa1-9e86-51cfd16f8025\") "
Sep 15 12:06:49.276225 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:49.276198    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de48197c-e0dc-4aa1-9e86-51cfd16f8025-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de48197c-e0dc-4aa1-9e86-51cfd16f8025" (UID: "de48197c-e0dc-4aa1-9e86-51cfd16f8025"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:06:49.377719 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:49.377625    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de48197c-e0dc-4aa1-9e86-51cfd16f8025-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:06:50.126057 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:50.125975    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6"
Sep 15 12:06:50.126057 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:50.126013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6" event={"ID":"de48197c-e0dc-4aa1-9e86-51cfd16f8025","Type":"ContainerDied","Data":"38da8da15088c78489765a293517f280f94d416208d514b7441956e856a2e461"}
Sep 15 12:06:50.126237 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:50.126101    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38da8da15088c78489765a293517f280f94d416208d514b7441956e856a2e461"
Sep 15 12:06:51.508643 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:51.508603    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"]
Sep 15 12:06:51.514519 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:51.514488    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"]
Sep 15 12:06:51.528549 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:51.528525    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"]
Sep 15 12:06:51.538599 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:51.538570    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"]
Sep 15 12:06:51.546782 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:06:51.546757    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"]
Sep 15 12:07:02.406476 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:02.406396    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-29d630-29824560-wpv5n"]
Sep 15 12:07:02.409017 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:02.408985    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-29d630-29824560-wpv5n"]
Sep 15 12:07:02.639938 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:02.639895    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6"]
Sep 15 12:07:02.645534 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:02.645506    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-29d630-7bhl6"]
Sep 15 12:07:03.730556 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:03.730525    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67547c19-52b2-4be3-8c21-7aa2253b647b" path="/var/lib/kubelet/pods/67547c19-52b2-4be3-8c21-7aa2253b647b/volumes"
Sep 15 12:07:03.730953 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:03.730859    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de48197c-e0dc-4aa1-9e86-51cfd16f8025" path="/var/lib/kubelet/pods/de48197c-e0dc-4aa1-9e86-51cfd16f8025/volumes"
Sep 15 12:07:06.528947 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.528893    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f" podUID="a3399618-3c99-46fb-80aa-f18a1558ef5b" containerName="main" containerID="cri-o://20bc015aa867bf610e8eac493e6ae6067fd53944b564b47be0c8bd919dbd8f5a" gracePeriod=45
Sep 15 12:07:06.534212 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.534167    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz" podUID="86e2c432-664d-437e-ab86-d91e2a0b3be5" containerName="main" containerID="cri-o://7f12ef8f0955d31e5896f7087bfff8aa31f1cb0ce8ca0081c64fe2bf0ffca80a" gracePeriod=45
Sep 15 12:07:06.547572 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.547533    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx" podUID="5bbe8724-a854-4772-9dc7-f70be0953337" containerName="main" containerID="cri-o://cec3297412c564acf8a0997bf720f99f1473493df31f5a8c44dced7e6d30a7cc" gracePeriod=45
Sep 15 12:07:06.559244 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.559217    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl" podUID="6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe" containerName="main" containerID="cri-o://425ceef44763354af4884b0662449a84f46fa7b837a1b0a394028c5694601fcb" gracePeriod=45
Sep 15 12:07:06.564602 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.564571    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb" podUID="0767ba87-5a08-4c34-8df4-dbffd98dbb75" containerName="main" containerID="cri-o://339f9b1e29775abef418c7595b9ed61756f7e192e248fc28cb62c61b930beb1e" gracePeriod=45
Sep 15 12:07:06.819737 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.819711    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:07:06.886921 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.886866    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:07:06.900509 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.900483    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:07:06.919461 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.919435    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 12:07:06.933184 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.933157    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 12:07:06.949456 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.949428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-model-cache\") pod \"86e2c432-664d-437e-ab86-d91e2a0b3be5\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") "
Sep 15 12:07:06.949578 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.949483    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-home\") pod \"86e2c432-664d-437e-ab86-d91e2a0b3be5\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") "
Sep 15 12:07:06.949578 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.949530    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-tmp-dir\") pod \"86e2c432-664d-437e-ab86-d91e2a0b3be5\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") "
Sep 15 12:07:06.949578 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.949563    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e2c432-664d-437e-ab86-d91e2a0b3be5-tls-certs\") pod \"86e2c432-664d-437e-ab86-d91e2a0b3be5\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") "
Sep 15 12:07:06.949770 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.949579    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-dshm\") pod \"86e2c432-664d-437e-ab86-d91e2a0b3be5\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") "
Sep 15 12:07:06.949770 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.949649    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvtl4\" (UniqueName: \"kubernetes.io/projected/86e2c432-664d-437e-ab86-d91e2a0b3be5-kube-api-access-jvtl4\") pod \"86e2c432-664d-437e-ab86-d91e2a0b3be5\" (UID: \"86e2c432-664d-437e-ab86-d91e2a0b3be5\") "
Sep 15 12:07:06.949770 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.949718    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-model-cache" (OuterVolumeSpecName: "model-cache") pod "86e2c432-664d-437e-ab86-d91e2a0b3be5" (UID: "86e2c432-664d-437e-ab86-d91e2a0b3be5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:06.949955 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.949810    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-home" (OuterVolumeSpecName: "home") pod "86e2c432-664d-437e-ab86-d91e2a0b3be5" (UID: "86e2c432-664d-437e-ab86-d91e2a0b3be5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:06.949955 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.949894    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86e2c432-664d-437e-ab86-d91e2a0b3be5" (UID: "86e2c432-664d-437e-ab86-d91e2a0b3be5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:06.950113 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.950095    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:06.950179 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.950119    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:06.950179 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.950136    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:06.952456 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.952431    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e2c432-664d-437e-ab86-d91e2a0b3be5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86e2c432-664d-437e-ab86-d91e2a0b3be5" (UID: "86e2c432-664d-437e-ab86-d91e2a0b3be5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:07:06.952557 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.952431    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-dshm" (OuterVolumeSpecName: "dshm") pod "86e2c432-664d-437e-ab86-d91e2a0b3be5" (UID: "86e2c432-664d-437e-ab86-d91e2a0b3be5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:06.952557 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:06.952491    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e2c432-664d-437e-ab86-d91e2a0b3be5-kube-api-access-jvtl4" (OuterVolumeSpecName: "kube-api-access-jvtl4") pod "86e2c432-664d-437e-ab86-d91e2a0b3be5" (UID: "86e2c432-664d-437e-ab86-d91e2a0b3be5"). InnerVolumeSpecName "kube-api-access-jvtl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:07:07.050539 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050502    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-home\") pod \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") "
Sep 15 12:07:07.050539 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050542    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-home\") pod \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") "
Sep 15 12:07:07.050797 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050566    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-tmp-dir\") pod \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") "
Sep 15 12:07:07.050797 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050589    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vb6h\" (UniqueName: \"kubernetes.io/projected/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-kube-api-access-9vb6h\") pod \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") "
Sep 15 12:07:07.050797 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050612    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-dshm\") pod \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") "
Sep 15 12:07:07.050797 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050629    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-dshm\") pod \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") "
Sep 15 12:07:07.050797 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trrfq\" (UniqueName: \"kubernetes.io/projected/0767ba87-5a08-4c34-8df4-dbffd98dbb75-kube-api-access-trrfq\") pod \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") "
Sep 15 12:07:07.050797 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050685    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zplb5\" (UniqueName: \"kubernetes.io/projected/a3399618-3c99-46fb-80aa-f18a1558ef5b-kube-api-access-zplb5\") pod \"a3399618-3c99-46fb-80aa-f18a1558ef5b\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") "
Sep 15 12:07:07.050797 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050708    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zblrb\" (UniqueName: \"kubernetes.io/projected/5bbe8724-a854-4772-9dc7-f70be0953337-kube-api-access-zblrb\") pod \"5bbe8724-a854-4772-9dc7-f70be0953337\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") "
Sep 15 12:07:07.050797 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050755    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3399618-3c99-46fb-80aa-f18a1558ef5b-tls-certs\") pod \"a3399618-3c99-46fb-80aa-f18a1558ef5b\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") "
Sep 15 12:07:07.050797 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050784    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-dshm\") pod \"a3399618-3c99-46fb-80aa-f18a1558ef5b\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") "
Sep 15 12:07:07.051296 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050806    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bbe8724-a854-4772-9dc7-f70be0953337-tls-certs\") pod \"5bbe8724-a854-4772-9dc7-f70be0953337\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") "
Sep 15 12:07:07.051296 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050817    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0767ba87-5a08-4c34-8df4-dbffd98dbb75" (UID: "0767ba87-5a08-4c34-8df4-dbffd98dbb75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.051296 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050836    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-tls-certs\") pod \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") "
Sep 15 12:07:07.051296 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.050827    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-home" (OuterVolumeSpecName: "home") pod "0767ba87-5a08-4c34-8df4-dbffd98dbb75" (UID: "0767ba87-5a08-4c34-8df4-dbffd98dbb75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.051503 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.051338    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-home" (OuterVolumeSpecName: "home") pod "6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe" (UID: "6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.053431 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.053364    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-dshm" (OuterVolumeSpecName: "dshm") pod "6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe" (UID: "6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.053431 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.053366    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe" (UID: "6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:07:07.053623 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.053471    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0767ba87-5a08-4c34-8df4-dbffd98dbb75-kube-api-access-trrfq" (OuterVolumeSpecName: "kube-api-access-trrfq") pod "0767ba87-5a08-4c34-8df4-dbffd98dbb75" (UID: "0767ba87-5a08-4c34-8df4-dbffd98dbb75"). InnerVolumeSpecName "kube-api-access-trrfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:07:07.053623 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.053559    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3399618-3c99-46fb-80aa-f18a1558ef5b-kube-api-access-zplb5" (OuterVolumeSpecName: "kube-api-access-zplb5") pod "a3399618-3c99-46fb-80aa-f18a1558ef5b" (UID: "a3399618-3c99-46fb-80aa-f18a1558ef5b"). InnerVolumeSpecName "kube-api-access-zplb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:07:07.053623 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.053612    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-model-cache\") pod \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") "
Sep 15 12:07:07.053770 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.053651    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0767ba87-5a08-4c34-8df4-dbffd98dbb75-tls-certs\") pod \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") "
Sep 15 12:07:07.053770 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.053691    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-tmp-dir\") pod \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\" (UID: \"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe\") "
Sep 15 12:07:07.053770 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.053719    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-tmp-dir\") pod \"a3399618-3c99-46fb-80aa-f18a1558ef5b\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") "
Sep 15 12:07:07.053770 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.053745    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-dshm\") pod \"5bbe8724-a854-4772-9dc7-f70be0953337\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") "
Sep 15 12:07:07.053970 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.053779    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-model-cache\") pod \"5bbe8724-a854-4772-9dc7-f70be0953337\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") "
Sep 15 12:07:07.053970 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.053806    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbe8724-a854-4772-9dc7-f70be0953337-kube-api-access-zblrb" (OuterVolumeSpecName: "kube-api-access-zblrb") pod "5bbe8724-a854-4772-9dc7-f70be0953337" (UID: "5bbe8724-a854-4772-9dc7-f70be0953337"). InnerVolumeSpecName "kube-api-access-zblrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:07:07.053970 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.053817    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-model-cache\") pod \"a3399618-3c99-46fb-80aa-f18a1558ef5b\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") "
Sep 15 12:07:07.053970 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.053891    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-model-cache\") pod \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\" (UID: \"0767ba87-5a08-4c34-8df4-dbffd98dbb75\") "
Sep 15 12:07:07.053970 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.053937    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-home\") pod \"5bbe8724-a854-4772-9dc7-f70be0953337\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") "
Sep 15 12:07:07.054179 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054034    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-tmp-dir\") pod \"5bbe8724-a854-4772-9dc7-f70be0953337\" (UID: \"5bbe8724-a854-4772-9dc7-f70be0953337\") "
Sep 15 12:07:07.054179 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054058    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-home\") pod \"a3399618-3c99-46fb-80aa-f18a1558ef5b\" (UID: \"a3399618-3c99-46fb-80aa-f18a1558ef5b\") "
Sep 15 12:07:07.054179 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054077    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe" (UID: "6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.054330 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054289    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-home" (OuterVolumeSpecName: "home") pod "a3399618-3c99-46fb-80aa-f18a1558ef5b" (UID: "a3399618-3c99-46fb-80aa-f18a1558ef5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.054664 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054446    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3399618-3c99-46fb-80aa-f18a1558ef5b" (UID: "a3399618-3c99-46fb-80aa-f18a1558ef5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.054664 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054539    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3399618-3c99-46fb-80aa-f18a1558ef5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3399618-3c99-46fb-80aa-f18a1558ef5b" (UID: "a3399618-3c99-46fb-80aa-f18a1558ef5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:07:07.054664 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054609    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvtl4\" (UniqueName: \"kubernetes.io/projected/86e2c432-664d-437e-ab86-d91e2a0b3be5-kube-api-access-jvtl4\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.054664 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054614    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-kube-api-access-9vb6h" (OuterVolumeSpecName: "kube-api-access-9vb6h") pod "6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe" (UID: "6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe"). InnerVolumeSpecName "kube-api-access-9vb6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:07:07.054664 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054630    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.054664 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054647    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.054664 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054660    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054672    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054685    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054697    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trrfq\" (UniqueName: \"kubernetes.io/projected/0767ba87-5a08-4c34-8df4-dbffd98dbb75-kube-api-access-trrfq\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054710    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zplb5\" (UniqueName: \"kubernetes.io/projected/a3399618-3c99-46fb-80aa-f18a1558ef5b-kube-api-access-zplb5\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054733    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zblrb\" (UniqueName: \"kubernetes.io/projected/5bbe8724-a854-4772-9dc7-f70be0953337-kube-api-access-zblrb\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054746    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054759    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054765    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe" (UID: "6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054771    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054829    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e2c432-664d-437e-ab86-d91e2a0b3be5-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054839    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-model-cache" (OuterVolumeSpecName: "model-cache") pod "0767ba87-5a08-4c34-8df4-dbffd98dbb75" (UID: "0767ba87-5a08-4c34-8df4-dbffd98dbb75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.054847    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e2c432-664d-437e-ab86-d91e2a0b3be5-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.055052    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-home" (OuterVolumeSpecName: "home") pod "5bbe8724-a854-4772-9dc7-f70be0953337" (UID: "5bbe8724-a854-4772-9dc7-f70be0953337"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.055071    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-model-cache" (OuterVolumeSpecName: "model-cache") pod "5bbe8724-a854-4772-9dc7-f70be0953337" (UID: "5bbe8724-a854-4772-9dc7-f70be0953337"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.055096    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-dshm" (OuterVolumeSpecName: "dshm") pod "0767ba87-5a08-4c34-8df4-dbffd98dbb75" (UID: "0767ba87-5a08-4c34-8df4-dbffd98dbb75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.055110    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5bbe8724-a854-4772-9dc7-f70be0953337" (UID: "5bbe8724-a854-4772-9dc7-f70be0953337"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.055522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.055297    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3399618-3c99-46fb-80aa-f18a1558ef5b" (UID: "a3399618-3c99-46fb-80aa-f18a1558ef5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.056428 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.055575    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-dshm" (OuterVolumeSpecName: "dshm") pod "a3399618-3c99-46fb-80aa-f18a1558ef5b" (UID: "a3399618-3c99-46fb-80aa-f18a1558ef5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.056428 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.055662    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bbe8724-a854-4772-9dc7-f70be0953337-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bbe8724-a854-4772-9dc7-f70be0953337" (UID: "5bbe8724-a854-4772-9dc7-f70be0953337"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:07:07.056557 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.056541    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-dshm" (OuterVolumeSpecName: "dshm") pod "5bbe8724-a854-4772-9dc7-f70be0953337" (UID: "5bbe8724-a854-4772-9dc7-f70be0953337"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:07.056778 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.056764    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0767ba87-5a08-4c34-8df4-dbffd98dbb75-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0767ba87-5a08-4c34-8df4-dbffd98dbb75" (UID: "0767ba87-5a08-4c34-8df4-dbffd98dbb75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:07:07.155588 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.155550    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vb6h\" (UniqueName: \"kubernetes.io/projected/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-kube-api-access-9vb6h\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.155588 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.155582    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.155588 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.155592    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3399618-3c99-46fb-80aa-f18a1558ef5b-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.155830 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.155602    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.155830 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.155612    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bbe8724-a854-4772-9dc7-f70be0953337-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.155830 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.155619    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.155830 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.155627    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0767ba87-5a08-4c34-8df4-dbffd98dbb75-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.155830 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.155634    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.155830 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.155641    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.155830 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.155648    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3399618-3c99-46fb-80aa-f18a1558ef5b-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.155830 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.155656    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0767ba87-5a08-4c34-8df4-dbffd98dbb75-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.155830 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.155662    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.155830 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.155669    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bbe8724-a854-4772-9dc7-f70be0953337-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:07.233582 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.233547    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3399618-3c99-46fb-80aa-f18a1558ef5b" containerID="20bc015aa867bf610e8eac493e6ae6067fd53944b564b47be0c8bd919dbd8f5a" exitCode=0
Sep 15 12:07:07.233750 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.233599    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"
Sep 15 12:07:07.233750 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.233638    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f" event={"ID":"a3399618-3c99-46fb-80aa-f18a1558ef5b","Type":"ContainerDied","Data":"20bc015aa867bf610e8eac493e6ae6067fd53944b564b47be0c8bd919dbd8f5a"}
Sep 15 12:07:07.233750 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.233696    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f" event={"ID":"a3399618-3c99-46fb-80aa-f18a1558ef5b","Type":"ContainerDied","Data":"68dc78da6fc860e9c4783b54e81df75821ee3e5c8528eb1df30104c1951685a7"}
Sep 15 12:07:07.233750 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.233723    2580 scope.go:117] "RemoveContainer" containerID="20bc015aa867bf610e8eac493e6ae6067fd53944b564b47be0c8bd919dbd8f5a"
Sep 15 12:07:07.235055 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.235034    2580 generic.go:356] "Generic (PLEG): container finished" podID="0767ba87-5a08-4c34-8df4-dbffd98dbb75" containerID="339f9b1e29775abef418c7595b9ed61756f7e192e248fc28cb62c61b930beb1e" exitCode=0
Sep 15 12:07:07.235188 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.235079    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"
Sep 15 12:07:07.235188 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.235099    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb" event={"ID":"0767ba87-5a08-4c34-8df4-dbffd98dbb75","Type":"ContainerDied","Data":"339f9b1e29775abef418c7595b9ed61756f7e192e248fc28cb62c61b930beb1e"}
Sep 15 12:07:07.235188 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.235151    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb" event={"ID":"0767ba87-5a08-4c34-8df4-dbffd98dbb75","Type":"ContainerDied","Data":"409a31e78a4b8c1a61f018db3a78cecc6431c3bdf63411806200f157b3403b25"}
Sep 15 12:07:07.236601 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.236576    2580 generic.go:356] "Generic (PLEG): container finished" podID="5bbe8724-a854-4772-9dc7-f70be0953337" containerID="cec3297412c564acf8a0997bf720f99f1473493df31f5a8c44dced7e6d30a7cc" exitCode=0
Sep 15 12:07:07.236744 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.236626    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"
Sep 15 12:07:07.236744 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.236656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx" event={"ID":"5bbe8724-a854-4772-9dc7-f70be0953337","Type":"ContainerDied","Data":"cec3297412c564acf8a0997bf720f99f1473493df31f5a8c44dced7e6d30a7cc"}
Sep 15 12:07:07.236744 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.236704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx" event={"ID":"5bbe8724-a854-4772-9dc7-f70be0953337","Type":"ContainerDied","Data":"452ba76f0ba7232d604a971db9937d22bf8d0a48a1c6ae2901847073e0fd95f8"}
Sep 15 12:07:07.238623 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.238587    2580 generic.go:356] "Generic (PLEG): container finished" podID="6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe" containerID="425ceef44763354af4884b0662449a84f46fa7b837a1b0a394028c5694601fcb" exitCode=0
Sep 15 12:07:07.238623 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.238619    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"
Sep 15 12:07:07.239298 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.238670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl" event={"ID":"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe","Type":"ContainerDied","Data":"425ceef44763354af4884b0662449a84f46fa7b837a1b0a394028c5694601fcb"}
Sep 15 12:07:07.239298 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.238700    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl" event={"ID":"6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe","Type":"ContainerDied","Data":"c8177364c67a3f43a8b65f2804f83e6cdbdd4893c38a3abd06a37384d8a8f790"}
Sep 15 12:07:07.242955 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.242931    2580 generic.go:356] "Generic (PLEG): container finished" podID="86e2c432-664d-437e-ab86-d91e2a0b3be5" containerID="7f12ef8f0955d31e5896f7087bfff8aa31f1cb0ce8ca0081c64fe2bf0ffca80a" exitCode=0
Sep 15 12:07:07.243073 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.242996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz" event={"ID":"86e2c432-664d-437e-ab86-d91e2a0b3be5","Type":"ContainerDied","Data":"7f12ef8f0955d31e5896f7087bfff8aa31f1cb0ce8ca0081c64fe2bf0ffca80a"}
Sep 15 12:07:07.243073 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.243024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz" event={"ID":"86e2c432-664d-437e-ab86-d91e2a0b3be5","Type":"ContainerDied","Data":"ae12e284faed9b98e1b7d407a4f9ffbd751285a02cd6e5fd42e33ad3f550ca7e"}
Sep 15 12:07:07.243242 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.243082    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"
Sep 15 12:07:07.245852 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.245828    2580 scope.go:117] "RemoveContainer" containerID="20bc015aa867bf610e8eac493e6ae6067fd53944b564b47be0c8bd919dbd8f5a"
Sep 15 12:07:07.246489 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:07:07.246215    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20bc015aa867bf610e8eac493e6ae6067fd53944b564b47be0c8bd919dbd8f5a\": container with ID starting with 20bc015aa867bf610e8eac493e6ae6067fd53944b564b47be0c8bd919dbd8f5a not found: ID does not exist" containerID="20bc015aa867bf610e8eac493e6ae6067fd53944b564b47be0c8bd919dbd8f5a"
Sep 15 12:07:07.246489 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.246247    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20bc015aa867bf610e8eac493e6ae6067fd53944b564b47be0c8bd919dbd8f5a"} err="failed to get container status \"20bc015aa867bf610e8eac493e6ae6067fd53944b564b47be0c8bd919dbd8f5a\": rpc error: code = NotFound desc = could not find container \"20bc015aa867bf610e8eac493e6ae6067fd53944b564b47be0c8bd919dbd8f5a\": container with ID starting with 20bc015aa867bf610e8eac493e6ae6067fd53944b564b47be0c8bd919dbd8f5a not found: ID does not exist"
Sep 15 12:07:07.246489 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.246272    2580 scope.go:117] "RemoveContainer" containerID="339f9b1e29775abef418c7595b9ed61756f7e192e248fc28cb62c61b930beb1e"
Sep 15 12:07:07.258255 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.258237    2580 scope.go:117] "RemoveContainer" containerID="339f9b1e29775abef418c7595b9ed61756f7e192e248fc28cb62c61b930beb1e"
Sep 15 12:07:07.258524 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:07:07.258503    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339f9b1e29775abef418c7595b9ed61756f7e192e248fc28cb62c61b930beb1e\": container with ID starting with 339f9b1e29775abef418c7595b9ed61756f7e192e248fc28cb62c61b930beb1e not found: ID does not exist" containerID="339f9b1e29775abef418c7595b9ed61756f7e192e248fc28cb62c61b930beb1e"
Sep 15 12:07:07.258574 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.258530    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339f9b1e29775abef418c7595b9ed61756f7e192e248fc28cb62c61b930beb1e"} err="failed to get container status \"339f9b1e29775abef418c7595b9ed61756f7e192e248fc28cb62c61b930beb1e\": rpc error: code = NotFound desc = could not find container \"339f9b1e29775abef418c7595b9ed61756f7e192e248fc28cb62c61b930beb1e\": container with ID starting with 339f9b1e29775abef418c7595b9ed61756f7e192e248fc28cb62c61b930beb1e not found: ID does not exist"
Sep 15 12:07:07.258574 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.258546    2580 scope.go:117] "RemoveContainer" containerID="cec3297412c564acf8a0997bf720f99f1473493df31f5a8c44dced7e6d30a7cc"
Sep 15 12:07:07.266304 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.266282    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"]
Sep 15 12:07:07.269459 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.268723    2580 scope.go:117] "RemoveContainer" containerID="cec3297412c564acf8a0997bf720f99f1473493df31f5a8c44dced7e6d30a7cc"
Sep 15 12:07:07.269459 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:07:07.269363    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec3297412c564acf8a0997bf720f99f1473493df31f5a8c44dced7e6d30a7cc\": container with ID starting with cec3297412c564acf8a0997bf720f99f1473493df31f5a8c44dced7e6d30a7cc not found: ID does not exist" containerID="cec3297412c564acf8a0997bf720f99f1473493df31f5a8c44dced7e6d30a7cc"
Sep 15 12:07:07.269459 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.269392    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec3297412c564acf8a0997bf720f99f1473493df31f5a8c44dced7e6d30a7cc"} err="failed to get container status \"cec3297412c564acf8a0997bf720f99f1473493df31f5a8c44dced7e6d30a7cc\": rpc error: code = NotFound desc = could not find container \"cec3297412c564acf8a0997bf720f99f1473493df31f5a8c44dced7e6d30a7cc\": container with ID starting with cec3297412c564acf8a0997bf720f99f1473493df31f5a8c44dced7e6d30a7cc not found: ID does not exist"
Sep 15 12:07:07.269459 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.269415    2580 scope.go:117] "RemoveContainer" containerID="425ceef44763354af4884b0662449a84f46fa7b837a1b0a394028c5694601fcb"
Sep 15 12:07:07.271961 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.271940    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/embedding-w0-29d630-kserve-5569b4d6d-96hmx"]
Sep 15 12:07:07.279312 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.279292    2580 scope.go:117] "RemoveContainer" containerID="425ceef44763354af4884b0662449a84f46fa7b837a1b0a394028c5694601fcb"
Sep 15 12:07:07.279584 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:07:07.279566    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425ceef44763354af4884b0662449a84f46fa7b837a1b0a394028c5694601fcb\": container with ID starting with 425ceef44763354af4884b0662449a84f46fa7b837a1b0a394028c5694601fcb not found: ID does not exist" containerID="425ceef44763354af4884b0662449a84f46fa7b837a1b0a394028c5694601fcb"
Sep 15 12:07:07.279651 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.279590    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425ceef44763354af4884b0662449a84f46fa7b837a1b0a394028c5694601fcb"} err="failed to get container status \"425ceef44763354af4884b0662449a84f46fa7b837a1b0a394028c5694601fcb\": rpc error: code = NotFound desc = could not find container \"425ceef44763354af4884b0662449a84f46fa7b837a1b0a394028c5694601fcb\": container with ID starting with 425ceef44763354af4884b0662449a84f46fa7b837a1b0a394028c5694601fcb not found: ID does not exist"
Sep 15 12:07:07.279651 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.279605    2580 scope.go:117] "RemoveContainer" containerID="7f12ef8f0955d31e5896f7087bfff8aa31f1cb0ce8ca0081c64fe2bf0ffca80a"
Sep 15 12:07:07.285326 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.285303    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"]
Sep 15 12:07:07.288769 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.288748    2580 scope.go:117] "RemoveContainer" containerID="7f12ef8f0955d31e5896f7087bfff8aa31f1cb0ce8ca0081c64fe2bf0ffca80a"
Sep 15 12:07:07.289087 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:07:07.289062    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f12ef8f0955d31e5896f7087bfff8aa31f1cb0ce8ca0081c64fe2bf0ffca80a\": container with ID starting with 7f12ef8f0955d31e5896f7087bfff8aa31f1cb0ce8ca0081c64fe2bf0ffca80a not found: ID does not exist" containerID="7f12ef8f0955d31e5896f7087bfff8aa31f1cb0ce8ca0081c64fe2bf0ffca80a"
Sep 15 12:07:07.289180 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.289096    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f12ef8f0955d31e5896f7087bfff8aa31f1cb0ce8ca0081c64fe2bf0ffca80a"} err="failed to get container status \"7f12ef8f0955d31e5896f7087bfff8aa31f1cb0ce8ca0081c64fe2bf0ffca80a\": rpc error: code = NotFound desc = could not find container \"7f12ef8f0955d31e5896f7087bfff8aa31f1cb0ce8ca0081c64fe2bf0ffca80a\": container with ID starting with 7f12ef8f0955d31e5896f7087bfff8aa31f1cb0ce8ca0081c64fe2bf0ffca80a not found: ID does not exist"
Sep 15 12:07:07.289245 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.289230    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/simulator-w0-29d630-kserve-7c5d9c975c-v98wb"]
Sep 15 12:07:07.300020 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.299997    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"]
Sep 15 12:07:07.304314 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.304263    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/distinct-2-w0-29d630-kserve-6bb959d4d-6rt9f"]
Sep 15 12:07:07.317202 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.317177    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"]
Sep 15 12:07:07.321382 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.321361    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/premium-w0-29d630-kserve-d4f876c56-xqbzl"]
Sep 15 12:07:07.332119 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.332095    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"]
Sep 15 12:07:07.338540 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.338520    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-29d630/distinct-w0-29d630-kserve-857ddc94b7-nhrbz"]
Sep 15 12:07:07.731020 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.730983    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0767ba87-5a08-4c34-8df4-dbffd98dbb75" path="/var/lib/kubelet/pods/0767ba87-5a08-4c34-8df4-dbffd98dbb75/volumes"
Sep 15 12:07:07.731423 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.731408    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bbe8724-a854-4772-9dc7-f70be0953337" path="/var/lib/kubelet/pods/5bbe8724-a854-4772-9dc7-f70be0953337/volumes"
Sep 15 12:07:07.731766 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.731754    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe" path="/var/lib/kubelet/pods/6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe/volumes"
Sep 15 12:07:07.732438 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.732422    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86e2c432-664d-437e-ab86-d91e2a0b3be5" path="/var/lib/kubelet/pods/86e2c432-664d-437e-ab86-d91e2a0b3be5/volumes"
Sep 15 12:07:07.732799 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:07.732786    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3399618-3c99-46fb-80aa-f18a1558ef5b" path="/var/lib/kubelet/pods/a3399618-3c99-46fb-80aa-f18a1558ef5b/volumes"
Sep 15 12:07:09.528162 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:09.528124    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"]
Sep 15 12:07:09.528595 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:09.528396    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d" podUID="26fda038-f54a-4b23-84ff-419f2d7d0365" containerName="istio-proxy" containerID="cri-o://fb4ad45f91ed67060d8b001b5fdab8af349ebd28a83faac648fbefef59a06fd3" gracePeriod=30
Sep 15 12:07:14.774050 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.774026    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 12:07:14.924797 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.924770    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-credential-socket\") pod \"26fda038-f54a-4b23-84ff-419f2d7d0365\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") "
Sep 15 12:07:14.924979 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.924833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-workload-socket\") pod \"26fda038-f54a-4b23-84ff-419f2d7d0365\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") "
Sep 15 12:07:14.924979 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.924852    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-envoy\") pod \"26fda038-f54a-4b23-84ff-419f2d7d0365\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") "
Sep 15 12:07:14.924979 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.924895    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-data\") pod \"26fda038-f54a-4b23-84ff-419f2d7d0365\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") "
Sep 15 12:07:14.924979 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.924947    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ntdk\" (UniqueName: \"kubernetes.io/projected/26fda038-f54a-4b23-84ff-419f2d7d0365-kube-api-access-7ntdk\") pod \"26fda038-f54a-4b23-84ff-419f2d7d0365\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") "
Sep 15 12:07:14.924979 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.924978    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fda038-f54a-4b23-84ff-419f2d7d0365-istiod-ca-cert\") pod \"26fda038-f54a-4b23-84ff-419f2d7d0365\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") "
Sep 15 12:07:14.925243 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.924995    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-workload-certs\") pod \"26fda038-f54a-4b23-84ff-419f2d7d0365\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") "
Sep 15 12:07:14.925243 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.925016    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-token\") pod \"26fda038-f54a-4b23-84ff-419f2d7d0365\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") "
Sep 15 12:07:14.925243 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.925056    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-podinfo\") pod \"26fda038-f54a-4b23-84ff-419f2d7d0365\" (UID: \"26fda038-f54a-4b23-84ff-419f2d7d0365\") "
Sep 15 12:07:14.925243 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.925108    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "26fda038-f54a-4b23-84ff-419f2d7d0365" (UID: "26fda038-f54a-4b23-84ff-419f2d7d0365"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:14.925243 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.925126    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "26fda038-f54a-4b23-84ff-419f2d7d0365" (UID: "26fda038-f54a-4b23-84ff-419f2d7d0365"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:14.925507 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.925267    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-data" (OuterVolumeSpecName: "istio-data") pod "26fda038-f54a-4b23-84ff-419f2d7d0365" (UID: "26fda038-f54a-4b23-84ff-419f2d7d0365"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:14.925507 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.925335    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "26fda038-f54a-4b23-84ff-419f2d7d0365" (UID: "26fda038-f54a-4b23-84ff-419f2d7d0365"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:14.925507 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.925423    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-workload-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:14.925507 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.925423    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fda038-f54a-4b23-84ff-419f2d7d0365-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "26fda038-f54a-4b23-84ff-419f2d7d0365" (UID: "26fda038-f54a-4b23-84ff-419f2d7d0365"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:07:14.925507 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.925441    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-credential-socket\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:14.925507 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.925457    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-workload-socket\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:14.925507 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.925471    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-data\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:14.927283 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.927251    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "26fda038-f54a-4b23-84ff-419f2d7d0365" (UID: "26fda038-f54a-4b23-84ff-419f2d7d0365"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:07:14.927387 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.927315    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-token" (OuterVolumeSpecName: "istio-token") pod "26fda038-f54a-4b23-84ff-419f2d7d0365" (UID: "26fda038-f54a-4b23-84ff-419f2d7d0365"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:07:14.927532 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.927502    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "26fda038-f54a-4b23-84ff-419f2d7d0365" (UID: "26fda038-f54a-4b23-84ff-419f2d7d0365"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:14.927532 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:14.927514    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fda038-f54a-4b23-84ff-419f2d7d0365-kube-api-access-7ntdk" (OuterVolumeSpecName: "kube-api-access-7ntdk") pod "26fda038-f54a-4b23-84ff-419f2d7d0365" (UID: "26fda038-f54a-4b23-84ff-419f2d7d0365"). InnerVolumeSpecName "kube-api-access-7ntdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:07:15.026274 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.026252    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fda038-f54a-4b23-84ff-419f2d7d0365-istiod-ca-cert\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:15.026274 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.026274    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-token\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:15.026432 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.026282    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-podinfo\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:15.026432 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.026290    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fda038-f54a-4b23-84ff-419f2d7d0365-istio-envoy\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:15.026432 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.026298    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ntdk\" (UniqueName: \"kubernetes.io/projected/26fda038-f54a-4b23-84ff-419f2d7d0365-kube-api-access-7ntdk\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:07:15.298200 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.298112    2580 generic.go:356] "Generic (PLEG): container finished" podID="26fda038-f54a-4b23-84ff-419f2d7d0365" containerID="fb4ad45f91ed67060d8b001b5fdab8af349ebd28a83faac648fbefef59a06fd3" exitCode=0
Sep 15 12:07:15.298200 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.298156    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"
Sep 15 12:07:15.298403 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.298199    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d" event={"ID":"26fda038-f54a-4b23-84ff-419f2d7d0365","Type":"ContainerDied","Data":"fb4ad45f91ed67060d8b001b5fdab8af349ebd28a83faac648fbefef59a06fd3"}
Sep 15 12:07:15.298403 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.298256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d" event={"ID":"26fda038-f54a-4b23-84ff-419f2d7d0365","Type":"ContainerDied","Data":"3f485901cbd39ca95e656a1bc3fe2e63fbd80c56327372f4a7c92bfab93c754f"}
Sep 15 12:07:15.298403 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.298287    2580 scope.go:117] "RemoveContainer" containerID="fb4ad45f91ed67060d8b001b5fdab8af349ebd28a83faac648fbefef59a06fd3"
Sep 15 12:07:15.308748 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.308732    2580 scope.go:117] "RemoveContainer" containerID="35669b666b12877e06d96a8dd6acb91d79aa5e2e03e2561ec8875af4d1cb1d6c"
Sep 15 12:07:15.319671 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.319651    2580 scope.go:117] "RemoveContainer" containerID="fb4ad45f91ed67060d8b001b5fdab8af349ebd28a83faac648fbefef59a06fd3"
Sep 15 12:07:15.320059 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:07:15.320020    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4ad45f91ed67060d8b001b5fdab8af349ebd28a83faac648fbefef59a06fd3\": container with ID starting with fb4ad45f91ed67060d8b001b5fdab8af349ebd28a83faac648fbefef59a06fd3 not found: ID does not exist" containerID="fb4ad45f91ed67060d8b001b5fdab8af349ebd28a83faac648fbefef59a06fd3"
Sep 15 12:07:15.320163 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.320059    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4ad45f91ed67060d8b001b5fdab8af349ebd28a83faac648fbefef59a06fd3"} err="failed to get container status \"fb4ad45f91ed67060d8b001b5fdab8af349ebd28a83faac648fbefef59a06fd3\": rpc error: code = NotFound desc = could not find container \"fb4ad45f91ed67060d8b001b5fdab8af349ebd28a83faac648fbefef59a06fd3\": container with ID starting with fb4ad45f91ed67060d8b001b5fdab8af349ebd28a83faac648fbefef59a06fd3 not found: ID does not exist"
Sep 15 12:07:15.320163 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.320085    2580 scope.go:117] "RemoveContainer" containerID="35669b666b12877e06d96a8dd6acb91d79aa5e2e03e2561ec8875af4d1cb1d6c"
Sep 15 12:07:15.320401 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:07:15.320383    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35669b666b12877e06d96a8dd6acb91d79aa5e2e03e2561ec8875af4d1cb1d6c\": container with ID starting with 35669b666b12877e06d96a8dd6acb91d79aa5e2e03e2561ec8875af4d1cb1d6c not found: ID does not exist" containerID="35669b666b12877e06d96a8dd6acb91d79aa5e2e03e2561ec8875af4d1cb1d6c"
Sep 15 12:07:15.320468 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.320419    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35669b666b12877e06d96a8dd6acb91d79aa5e2e03e2561ec8875af4d1cb1d6c"} err="failed to get container status \"35669b666b12877e06d96a8dd6acb91d79aa5e2e03e2561ec8875af4d1cb1d6c\": rpc error: code = NotFound desc = could not find container \"35669b666b12877e06d96a8dd6acb91d79aa5e2e03e2561ec8875af4d1cb1d6c\": container with ID starting with 35669b666b12877e06d96a8dd6acb91d79aa5e2e03e2561ec8875af4d1cb1d6c not found: ID does not exist"
Sep 15 12:07:15.322388 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.322352    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"]
Sep 15 12:07:15.324667 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.324648    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-29d630-openshift-default-65f568fcd4-d5j2d"]
Sep 15 12:07:15.731649 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:15.731617    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26fda038-f54a-4b23-84ff-419f2d7d0365" path="/var/lib/kubelet/pods/26fda038-f54a-4b23-84ff-419f2d7d0365/volumes"
Sep 15 12:07:32.110400 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:32.110360    2580 scope.go:117] "RemoveContainer" containerID="8e44e5eccedafc16bbb433cec347a77b0407471aa232d0745a5be8d1c3d1f43c"
Sep 15 12:07:32.118950 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:07:32.118928    2580 scope.go:117] "RemoveContainer" containerID="f8e42c2c4c559139007eb6b034b19d8f516e46229e2f4bf38fffbc984fb78c24"
Sep 15 12:08:08.385255 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385217    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src"]
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385562    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26fda038-f54a-4b23-84ff-419f2d7d0365" containerName="istio-proxy"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385574    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="26fda038-f54a-4b23-84ff-419f2d7d0365" containerName="istio-proxy"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385582    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0767ba87-5a08-4c34-8df4-dbffd98dbb75" containerName="main"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385587    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0767ba87-5a08-4c34-8df4-dbffd98dbb75" containerName="main"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385599    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3399618-3c99-46fb-80aa-f18a1558ef5b" containerName="main"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385605    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3399618-3c99-46fb-80aa-f18a1558ef5b" containerName="main"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385614    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bbe8724-a854-4772-9dc7-f70be0953337" containerName="main"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385619    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bbe8724-a854-4772-9dc7-f70be0953337" containerName="main"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385626    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26fda038-f54a-4b23-84ff-419f2d7d0365" containerName="istio-proxy"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385632    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="26fda038-f54a-4b23-84ff-419f2d7d0365" containerName="istio-proxy"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385640    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe" containerName="main"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385645    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe" containerName="main"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385651    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de48197c-e0dc-4aa1-9e86-51cfd16f8025" containerName="revoke-keys"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385656    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="de48197c-e0dc-4aa1-9e86-51cfd16f8025" containerName="revoke-keys"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385663    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e2c432-664d-437e-ab86-d91e2a0b3be5" containerName="main"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385668    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e2c432-664d-437e-ab86-d91e2a0b3be5" containerName="main"
Sep 15 12:08:08.385713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385715    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cea3d4a-e390-4dcb-a8a9-3c34d8005fbe" containerName="main"
Sep 15 12:08:08.386253 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385723    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="26fda038-f54a-4b23-84ff-419f2d7d0365" containerName="istio-proxy"
Sep 15 12:08:08.386253 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385730    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0767ba87-5a08-4c34-8df4-dbffd98dbb75" containerName="main"
Sep 15 12:08:08.386253 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385737    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="26fda038-f54a-4b23-84ff-419f2d7d0365" containerName="istio-proxy"
Sep 15 12:08:08.386253 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385742    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="de48197c-e0dc-4aa1-9e86-51cfd16f8025" containerName="revoke-keys"
Sep 15 12:08:08.386253 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385749    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e2c432-664d-437e-ab86-d91e2a0b3be5" containerName="main"
Sep 15 12:08:08.386253 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385754    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3399618-3c99-46fb-80aa-f18a1558ef5b" containerName="main"
Sep 15 12:08:08.386253 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.385759    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bbe8724-a854-4772-9dc7-f70be0953337" containerName="main"
Sep 15 12:08:08.388373 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.388354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src"
Sep 15 12:08:08.391018 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.390997    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:08:08.391143 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.391000    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:08:08.396900 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.396842    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src"]
Sep 15 12:08:08.501648 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.501605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae2c82-2ef5-4ea4-8878-71c719d6868b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src\" (UID: \"f2ae2c82-2ef5-4ea4-8878-71c719d6868b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src"
Sep 15 12:08:08.602934 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.602867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae2c82-2ef5-4ea4-8878-71c719d6868b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src\" (UID: \"f2ae2c82-2ef5-4ea4-8878-71c719d6868b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src"
Sep 15 12:08:08.603482 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.603464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae2c82-2ef5-4ea4-8878-71c719d6868b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src\" (UID: \"f2ae2c82-2ef5-4ea4-8878-71c719d6868b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src"
Sep 15 12:08:08.701294 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.701261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src"
Sep 15 12:08:08.976517 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:08.974596    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src"]
Sep 15 12:08:08.978825 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:08:08.978796    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ae2c82_2ef5_4ea4_8878_71c719d6868b.slice/crio-6213b67c15d8de935b18cf56366303a432b08b8ab0b1c5414453d2d8a6e61238 WatchSource:0}: Error finding container 6213b67c15d8de935b18cf56366303a432b08b8ab0b1c5414453d2d8a6e61238: Status 404 returned error can't find the container with id 6213b67c15d8de935b18cf56366303a432b08b8ab0b1c5414453d2d8a6e61238
Sep 15 12:08:09.612282 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:09.612246    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src" event={"ID":"f2ae2c82-2ef5-4ea4-8878-71c719d6868b","Type":"ContainerStarted","Data":"f774ad5a5a0c035b55b504b13ce9ac254ea2fbdf17673d65e9bd05a317be65a2"}
Sep 15 12:08:09.612671 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:09.612306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src" event={"ID":"f2ae2c82-2ef5-4ea4-8878-71c719d6868b","Type":"ContainerStarted","Data":"6213b67c15d8de935b18cf56366303a432b08b8ab0b1c5414453d2d8a6e61238"}
Sep 15 12:08:09.629824 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:09.629767    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src" podStartSLOduration=1.6297505650000002 podStartE2EDuration="1.629750565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:08:09.627099225 +0000 UTC m=+1778.407670614" watchObservedRunningTime="2026-09-15 12:08:09.629750565 +0000 UTC m=+1778.410321952"
Sep 15 12:08:10.619680 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:10.619591    2580 generic.go:356] "Generic (PLEG): container finished" podID="f2ae2c82-2ef5-4ea4-8878-71c719d6868b" containerID="f774ad5a5a0c035b55b504b13ce9ac254ea2fbdf17673d65e9bd05a317be65a2" exitCode=0
Sep 15 12:08:10.619680 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:10.619662    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src" event={"ID":"f2ae2c82-2ef5-4ea4-8878-71c719d6868b","Type":"ContainerDied","Data":"f774ad5a5a0c035b55b504b13ce9ac254ea2fbdf17673d65e9bd05a317be65a2"}
Sep 15 12:08:11.754182 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:11.754146    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src"
Sep 15 12:08:11.837084 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:11.837048    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae2c82-2ef5-4ea4-8878-71c719d6868b-maas-api-service-ca\") pod \"f2ae2c82-2ef5-4ea4-8878-71c719d6868b\" (UID: \"f2ae2c82-2ef5-4ea4-8878-71c719d6868b\") "
Sep 15 12:08:11.837395 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:11.837370    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ae2c82-2ef5-4ea4-8878-71c719d6868b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f2ae2c82-2ef5-4ea4-8878-71c719d6868b" (UID: "f2ae2c82-2ef5-4ea4-8878-71c719d6868b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:08:11.938279 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:11.938247    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae2c82-2ef5-4ea4-8878-71c719d6868b-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:12.632999 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:12.632956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src" event={"ID":"f2ae2c82-2ef5-4ea4-8878-71c719d6868b","Type":"ContainerDied","Data":"6213b67c15d8de935b18cf56366303a432b08b8ab0b1c5414453d2d8a6e61238"}
Sep 15 12:08:12.632999 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:12.632990    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src"
Sep 15 12:08:12.633202 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:12.633047    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6213b67c15d8de935b18cf56366303a432b08b8ab0b1c5414453d2d8a6e61238"
Sep 15 12:08:13.243343 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:13.243305    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"]
Sep 15 12:08:13.248608 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:13.248574    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"]
Sep 15 12:08:13.257897 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:13.257857    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"]
Sep 15 12:08:13.270606 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:13.270581    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"]
Sep 15 12:08:24.065713 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:24.065639    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac76e5-29824560-fjvmh"]
Sep 15 12:08:24.071906 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:24.071265    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac76e5-29824560-fjvmh"]
Sep 15 12:08:24.342978 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:24.342889    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src"]
Sep 15 12:08:24.345018 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:24.344993    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac76e5-g2src"]
Sep 15 12:08:25.729886 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:25.729837    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6e97acc-8531-4a9a-b068-7425fc3a4434" path="/var/lib/kubelet/pods/a6e97acc-8531-4a9a-b068-7425fc3a4434/volumes"
Sep 15 12:08:25.730259 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:25.730179    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2ae2c82-2ef5-4ea4-8878-71c719d6868b" path="/var/lib/kubelet/pods/f2ae2c82-2ef5-4ea4-8878-71c719d6868b/volumes"
Sep 15 12:08:28.268352 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.268293    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h" podUID="df1da465-3751-4cf8-b8a8-e9faa35fd98f" containerName="main" containerID="cri-o://ea946eaee32b0636911616d3e087917edfd7df38737b6d351d7fdc205dd06a03" gracePeriod=45
Sep 15 12:08:28.279528 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.279491    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn" podUID="40091be4-fed1-4e6b-bded-82882eda3801" containerName="main" containerID="cri-o://d5b8167a0468026b170a6388e4d707ca2864083ecb1171ecf0b6907950980f0b" gracePeriod=45
Sep 15 12:08:28.288579 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.288549    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj" podUID="fb869d8e-73ba-4545-956a-e42501dd8ec9" containerName="main" containerID="cri-o://34152fee890eafb7a862ba12b76f5f82ef3be38701c14e8ae432a49e0e4f53a3" gracePeriod=45
Sep 15 12:08:28.578124 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.578087    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:08:28.580979 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.580960    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 12:08:28.584119 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.584101    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 12:08:28.656704 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.656659    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4" podUID="856006a0-813c-40e5-aca3-e6937e75a532" containerName="main" containerID="cri-o://34d915be02f327e5169fe2ed67f4130b33c6f4ecb9c07b0772dc1b985eba3193" gracePeriod=45
Sep 15 12:08:28.699572 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699548    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-model-cache\") pod \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") "
Sep 15 12:08:28.699674 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699584    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-tmp-dir\") pod \"40091be4-fed1-4e6b-bded-82882eda3801\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") "
Sep 15 12:08:28.699674 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699611    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-model-cache\") pod \"fb869d8e-73ba-4545-956a-e42501dd8ec9\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") "
Sep 15 12:08:28.699674 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699663    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-home\") pod \"40091be4-fed1-4e6b-bded-82882eda3801\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") "
Sep 15 12:08:28.699797 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699691    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb869d8e-73ba-4545-956a-e42501dd8ec9-tls-certs\") pod \"fb869d8e-73ba-4545-956a-e42501dd8ec9\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") "
Sep 15 12:08:28.699797 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699729    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-home\") pod \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") "
Sep 15 12:08:28.699797 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699756    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1da465-3751-4cf8-b8a8-e9faa35fd98f-tls-certs\") pod \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") "
Sep 15 12:08:28.699968 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699799    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw2z6\" (UniqueName: \"kubernetes.io/projected/fb869d8e-73ba-4545-956a-e42501dd8ec9-kube-api-access-cw2z6\") pod \"fb869d8e-73ba-4545-956a-e42501dd8ec9\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") "
Sep 15 12:08:28.699968 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699840    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-tmp-dir\") pod \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") "
Sep 15 12:08:28.699968 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699846    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-model-cache" (OuterVolumeSpecName: "model-cache") pod "df1da465-3751-4cf8-b8a8-e9faa35fd98f" (UID: "df1da465-3751-4cf8-b8a8-e9faa35fd98f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:28.699968 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699851    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40091be4-fed1-4e6b-bded-82882eda3801" (UID: "40091be4-fed1-4e6b-bded-82882eda3801"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:28.699968 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699902    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-dshm\") pod \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") "
Sep 15 12:08:28.699968 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb869d8e-73ba-4545-956a-e42501dd8ec9" (UID: "fb869d8e-73ba-4545-956a-e42501dd8ec9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:28.700292 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699953    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-home" (OuterVolumeSpecName: "home") pod "40091be4-fed1-4e6b-bded-82882eda3801" (UID: "40091be4-fed1-4e6b-bded-82882eda3801"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:28.700292 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.699995    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbplw\" (UniqueName: \"kubernetes.io/projected/40091be4-fed1-4e6b-bded-82882eda3801-kube-api-access-wbplw\") pod \"40091be4-fed1-4e6b-bded-82882eda3801\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") "
Sep 15 12:08:28.700292 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.700031    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40091be4-fed1-4e6b-bded-82882eda3801-tls-certs\") pod \"40091be4-fed1-4e6b-bded-82882eda3801\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") "
Sep 15 12:08:28.700292 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.700038    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-home" (OuterVolumeSpecName: "home") pod "df1da465-3751-4cf8-b8a8-e9faa35fd98f" (UID: "df1da465-3751-4cf8-b8a8-e9faa35fd98f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:28.700292 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.700060    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-dshm\") pod \"40091be4-fed1-4e6b-bded-82882eda3801\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") "
Sep 15 12:08:28.700292 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.700164    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df1da465-3751-4cf8-b8a8-e9faa35fd98f" (UID: "df1da465-3751-4cf8-b8a8-e9faa35fd98f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:28.700601 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.700425    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-home\") pod \"fb869d8e-73ba-4545-956a-e42501dd8ec9\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") "
Sep 15 12:08:28.700601 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.700461    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-dshm\") pod \"fb869d8e-73ba-4545-956a-e42501dd8ec9\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") "
Sep 15 12:08:28.700601 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.700512    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-tmp-dir\") pod \"fb869d8e-73ba-4545-956a-e42501dd8ec9\" (UID: \"fb869d8e-73ba-4545-956a-e42501dd8ec9\") "
Sep 15 12:08:28.700601 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.700542    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm429\" (UniqueName: \"kubernetes.io/projected/df1da465-3751-4cf8-b8a8-e9faa35fd98f-kube-api-access-pm429\") pod \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\" (UID: \"df1da465-3751-4cf8-b8a8-e9faa35fd98f\") "
Sep 15 12:08:28.700601 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.700570    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-model-cache\") pod \"40091be4-fed1-4e6b-bded-82882eda3801\" (UID: \"40091be4-fed1-4e6b-bded-82882eda3801\") "
Sep 15 12:08:28.701754 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.700949    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.701754 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.700972    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.701754 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.700985    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.701754 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.700997    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.701754 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.701010    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.701754 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.701022    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.701754 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.701197    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-model-cache" (OuterVolumeSpecName: "model-cache") pod "40091be4-fed1-4e6b-bded-82882eda3801" (UID: "40091be4-fed1-4e6b-bded-82882eda3801"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:28.701754 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.701356    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb869d8e-73ba-4545-956a-e42501dd8ec9" (UID: "fb869d8e-73ba-4545-956a-e42501dd8ec9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:28.701754 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.701533    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-home" (OuterVolumeSpecName: "home") pod "fb869d8e-73ba-4545-956a-e42501dd8ec9" (UID: "fb869d8e-73ba-4545-956a-e42501dd8ec9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:28.702419 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.702343    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb869d8e-73ba-4545-956a-e42501dd8ec9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb869d8e-73ba-4545-956a-e42501dd8ec9" (UID: "fb869d8e-73ba-4545-956a-e42501dd8ec9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:08:28.702419 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.702396    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-dshm" (OuterVolumeSpecName: "dshm") pod "df1da465-3751-4cf8-b8a8-e9faa35fd98f" (UID: "df1da465-3751-4cf8-b8a8-e9faa35fd98f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:28.702546 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.702493    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb869d8e-73ba-4545-956a-e42501dd8ec9-kube-api-access-cw2z6" (OuterVolumeSpecName: "kube-api-access-cw2z6") pod "fb869d8e-73ba-4545-956a-e42501dd8ec9" (UID: "fb869d8e-73ba-4545-956a-e42501dd8ec9"). InnerVolumeSpecName "kube-api-access-cw2z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:08:28.702610 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.702548    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df1da465-3751-4cf8-b8a8-e9faa35fd98f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df1da465-3751-4cf8-b8a8-e9faa35fd98f" (UID: "df1da465-3751-4cf8-b8a8-e9faa35fd98f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:08:28.702812 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.702793    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40091be4-fed1-4e6b-bded-82882eda3801-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40091be4-fed1-4e6b-bded-82882eda3801" (UID: "40091be4-fed1-4e6b-bded-82882eda3801"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:08:28.703231 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.703213    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-dshm" (OuterVolumeSpecName: "dshm") pod "40091be4-fed1-4e6b-bded-82882eda3801" (UID: "40091be4-fed1-4e6b-bded-82882eda3801"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:28.703334 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.703319    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40091be4-fed1-4e6b-bded-82882eda3801-kube-api-access-wbplw" (OuterVolumeSpecName: "kube-api-access-wbplw") pod "40091be4-fed1-4e6b-bded-82882eda3801" (UID: "40091be4-fed1-4e6b-bded-82882eda3801"). InnerVolumeSpecName "kube-api-access-wbplw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:08:28.703770 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.703751    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-dshm" (OuterVolumeSpecName: "dshm") pod "fb869d8e-73ba-4545-956a-e42501dd8ec9" (UID: "fb869d8e-73ba-4545-956a-e42501dd8ec9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:28.703985 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.703958    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1da465-3751-4cf8-b8a8-e9faa35fd98f-kube-api-access-pm429" (OuterVolumeSpecName: "kube-api-access-pm429") pod "df1da465-3751-4cf8-b8a8-e9faa35fd98f" (UID: "df1da465-3751-4cf8-b8a8-e9faa35fd98f"). InnerVolumeSpecName "kube-api-access-pm429". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:08:28.734238 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.734212    2580 generic.go:356] "Generic (PLEG): container finished" podID="40091be4-fed1-4e6b-bded-82882eda3801" containerID="d5b8167a0468026b170a6388e4d707ca2864083ecb1171ecf0b6907950980f0b" exitCode=0
Sep 15 12:08:28.734331 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.734250    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"
Sep 15 12:08:28.734331 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.734295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn" event={"ID":"40091be4-fed1-4e6b-bded-82882eda3801","Type":"ContainerDied","Data":"d5b8167a0468026b170a6388e4d707ca2864083ecb1171ecf0b6907950980f0b"}
Sep 15 12:08:28.734429 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.734337    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn" event={"ID":"40091be4-fed1-4e6b-bded-82882eda3801","Type":"ContainerDied","Data":"139a79dfec19dfd2eededfc0d044b9d8ab7ce35a025b7a1bdde87d6d3336141a"}
Sep 15 12:08:28.734429 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.734355    2580 scope.go:117] "RemoveContainer" containerID="d5b8167a0468026b170a6388e4d707ca2864083ecb1171ecf0b6907950980f0b"
Sep 15 12:08:28.736011 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.735992    2580 generic.go:356] "Generic (PLEG): container finished" podID="df1da465-3751-4cf8-b8a8-e9faa35fd98f" containerID="ea946eaee32b0636911616d3e087917edfd7df38737b6d351d7fdc205dd06a03" exitCode=0
Sep 15 12:08:28.736105 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.736037    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"
Sep 15 12:08:28.736105 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.736094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h" event={"ID":"df1da465-3751-4cf8-b8a8-e9faa35fd98f","Type":"ContainerDied","Data":"ea946eaee32b0636911616d3e087917edfd7df38737b6d351d7fdc205dd06a03"}
Sep 15 12:08:28.736222 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.736128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h" event={"ID":"df1da465-3751-4cf8-b8a8-e9faa35fd98f","Type":"ContainerDied","Data":"8d44720587cd80eeaf4c6f42d457ccace77f178ec1810191436283462acda837"}
Sep 15 12:08:28.742897 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.739057    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb869d8e-73ba-4545-956a-e42501dd8ec9" containerID="34152fee890eafb7a862ba12b76f5f82ef3be38701c14e8ae432a49e0e4f53a3" exitCode=0
Sep 15 12:08:28.742897 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.739254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj" event={"ID":"fb869d8e-73ba-4545-956a-e42501dd8ec9","Type":"ContainerDied","Data":"34152fee890eafb7a862ba12b76f5f82ef3be38701c14e8ae432a49e0e4f53a3"}
Sep 15 12:08:28.742897 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.739284    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj" event={"ID":"fb869d8e-73ba-4545-956a-e42501dd8ec9","Type":"ContainerDied","Data":"a15a071d90d034476d8902c7693e20e000d747a4f5567c5f48126dd3a6c645b3"}
Sep 15 12:08:28.742897 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.739352    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"
Sep 15 12:08:28.762108 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.762085    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"]
Sep 15 12:08:28.766635 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.766610    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/premium-w4-ac76e5-kserve-7f6fbb9d45-9pwbn"]
Sep 15 12:08:28.769660 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.769643    2580 scope.go:117] "RemoveContainer" containerID="d5b8167a0468026b170a6388e4d707ca2864083ecb1171ecf0b6907950980f0b"
Sep 15 12:08:28.770010 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:08:28.769988    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b8167a0468026b170a6388e4d707ca2864083ecb1171ecf0b6907950980f0b\": container with ID starting with d5b8167a0468026b170a6388e4d707ca2864083ecb1171ecf0b6907950980f0b not found: ID does not exist" containerID="d5b8167a0468026b170a6388e4d707ca2864083ecb1171ecf0b6907950980f0b"
Sep 15 12:08:28.770082 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.770016    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b8167a0468026b170a6388e4d707ca2864083ecb1171ecf0b6907950980f0b"} err="failed to get container status \"d5b8167a0468026b170a6388e4d707ca2864083ecb1171ecf0b6907950980f0b\": rpc error: code = NotFound desc = could not find container \"d5b8167a0468026b170a6388e4d707ca2864083ecb1171ecf0b6907950980f0b\": container with ID starting with d5b8167a0468026b170a6388e4d707ca2864083ecb1171ecf0b6907950980f0b not found: ID does not exist"
Sep 15 12:08:28.770082 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.770033    2580 scope.go:117] "RemoveContainer" containerID="ea946eaee32b0636911616d3e087917edfd7df38737b6d351d7fdc205dd06a03"
Sep 15 12:08:28.778268 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.778241    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"]
Sep 15 12:08:28.778574 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.778557    2580 scope.go:117] "RemoveContainer" containerID="ea946eaee32b0636911616d3e087917edfd7df38737b6d351d7fdc205dd06a03"
Sep 15 12:08:28.778862 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:08:28.778841    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea946eaee32b0636911616d3e087917edfd7df38737b6d351d7fdc205dd06a03\": container with ID starting with ea946eaee32b0636911616d3e087917edfd7df38737b6d351d7fdc205dd06a03 not found: ID does not exist" containerID="ea946eaee32b0636911616d3e087917edfd7df38737b6d351d7fdc205dd06a03"
Sep 15 12:08:28.778955 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.778890    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea946eaee32b0636911616d3e087917edfd7df38737b6d351d7fdc205dd06a03"} err="failed to get container status \"ea946eaee32b0636911616d3e087917edfd7df38737b6d351d7fdc205dd06a03\": rpc error: code = NotFound desc = could not find container \"ea946eaee32b0636911616d3e087917edfd7df38737b6d351d7fdc205dd06a03\": container with ID starting with ea946eaee32b0636911616d3e087917edfd7df38737b6d351d7fdc205dd06a03 not found: ID does not exist"
Sep 15 12:08:28.778955 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.778915    2580 scope.go:117] "RemoveContainer" containerID="34152fee890eafb7a862ba12b76f5f82ef3be38701c14e8ae432a49e0e4f53a3"
Sep 15 12:08:28.784513 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.784490    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/simulator-w4-ac76e5-kserve-84d48fb568-w9shj"]
Sep 15 12:08:28.789305 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.789286    2580 scope.go:117] "RemoveContainer" containerID="34152fee890eafb7a862ba12b76f5f82ef3be38701c14e8ae432a49e0e4f53a3"
Sep 15 12:08:28.789602 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:08:28.789575    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34152fee890eafb7a862ba12b76f5f82ef3be38701c14e8ae432a49e0e4f53a3\": container with ID starting with 34152fee890eafb7a862ba12b76f5f82ef3be38701c14e8ae432a49e0e4f53a3 not found: ID does not exist" containerID="34152fee890eafb7a862ba12b76f5f82ef3be38701c14e8ae432a49e0e4f53a3"
Sep 15 12:08:28.789672 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.789605    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34152fee890eafb7a862ba12b76f5f82ef3be38701c14e8ae432a49e0e4f53a3"} err="failed to get container status \"34152fee890eafb7a862ba12b76f5f82ef3be38701c14e8ae432a49e0e4f53a3\": rpc error: code = NotFound desc = could not find container \"34152fee890eafb7a862ba12b76f5f82ef3be38701c14e8ae432a49e0e4f53a3\": container with ID starting with 34152fee890eafb7a862ba12b76f5f82ef3be38701c14e8ae432a49e0e4f53a3 not found: ID does not exist"
Sep 15 12:08:28.795249 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.795214    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"]
Sep 15 12:08:28.797798 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.797773    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/distinct-w4-ac76e5-kserve-77896b84dd-nkf2h"]
Sep 15 12:08:28.801857 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.801803    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.801857 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.801830    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.801857 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.801843    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb869d8e-73ba-4545-956a-e42501dd8ec9-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.801857 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.801856    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm429\" (UniqueName: \"kubernetes.io/projected/df1da465-3751-4cf8-b8a8-e9faa35fd98f-kube-api-access-pm429\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.802140 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.801865    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.802140 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.801897    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb869d8e-73ba-4545-956a-e42501dd8ec9-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.802140 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.801904    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1da465-3751-4cf8-b8a8-e9faa35fd98f-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.802140 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.801912    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw2z6\" (UniqueName: \"kubernetes.io/projected/fb869d8e-73ba-4545-956a-e42501dd8ec9-kube-api-access-cw2z6\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.802140 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.801919    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df1da465-3751-4cf8-b8a8-e9faa35fd98f-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.802140 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.801927    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbplw\" (UniqueName: \"kubernetes.io/projected/40091be4-fed1-4e6b-bded-82882eda3801-kube-api-access-wbplw\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.802140 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.801937    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40091be4-fed1-4e6b-bded-82882eda3801-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.802140 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.801945    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40091be4-fed1-4e6b-bded-82882eda3801-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:28.897532 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:28.897503    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:08:29.003466 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.003435    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf72h\" (UniqueName: \"kubernetes.io/projected/856006a0-813c-40e5-aca3-e6937e75a532-kube-api-access-lf72h\") pod \"856006a0-813c-40e5-aca3-e6937e75a532\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") "
Sep 15 12:08:29.003652 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.003502    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-model-cache\") pod \"856006a0-813c-40e5-aca3-e6937e75a532\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") "
Sep 15 12:08:29.003652 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.003532    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-home\") pod \"856006a0-813c-40e5-aca3-e6937e75a532\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") "
Sep 15 12:08:29.003652 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.003547    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-tmp-dir\") pod \"856006a0-813c-40e5-aca3-e6937e75a532\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") "
Sep 15 12:08:29.003652 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.003614    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/856006a0-813c-40e5-aca3-e6937e75a532-tls-certs\") pod \"856006a0-813c-40e5-aca3-e6937e75a532\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") "
Sep 15 12:08:29.003841 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.003665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-dshm\") pod \"856006a0-813c-40e5-aca3-e6937e75a532\" (UID: \"856006a0-813c-40e5-aca3-e6937e75a532\") "
Sep 15 12:08:29.003841 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.003773    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-model-cache" (OuterVolumeSpecName: "model-cache") pod "856006a0-813c-40e5-aca3-e6937e75a532" (UID: "856006a0-813c-40e5-aca3-e6937e75a532"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:29.003841 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.003804    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-home" (OuterVolumeSpecName: "home") pod "856006a0-813c-40e5-aca3-e6937e75a532" (UID: "856006a0-813c-40e5-aca3-e6937e75a532"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:29.004041 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.003909    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "856006a0-813c-40e5-aca3-e6937e75a532" (UID: "856006a0-813c-40e5-aca3-e6937e75a532"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:29.004041 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.003983    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:29.004041 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.003997    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:29.004041 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.004010    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:29.005588 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.005567    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856006a0-813c-40e5-aca3-e6937e75a532-kube-api-access-lf72h" (OuterVolumeSpecName: "kube-api-access-lf72h") pod "856006a0-813c-40e5-aca3-e6937e75a532" (UID: "856006a0-813c-40e5-aca3-e6937e75a532"). InnerVolumeSpecName "kube-api-access-lf72h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:08:29.005666 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.005645    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-dshm" (OuterVolumeSpecName: "dshm") pod "856006a0-813c-40e5-aca3-e6937e75a532" (UID: "856006a0-813c-40e5-aca3-e6937e75a532"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:29.005752 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.005737    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856006a0-813c-40e5-aca3-e6937e75a532-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "856006a0-813c-40e5-aca3-e6937e75a532" (UID: "856006a0-813c-40e5-aca3-e6937e75a532"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:08:29.105218 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.105127    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/856006a0-813c-40e5-aca3-e6937e75a532-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:29.105218 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.105159    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/856006a0-813c-40e5-aca3-e6937e75a532-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:29.105218 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.105172    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf72h\" (UniqueName: \"kubernetes.io/projected/856006a0-813c-40e5-aca3-e6937e75a532-kube-api-access-lf72h\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:08:29.730255 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.730214    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40091be4-fed1-4e6b-bded-82882eda3801" path="/var/lib/kubelet/pods/40091be4-fed1-4e6b-bded-82882eda3801/volumes"
Sep 15 12:08:29.730804 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.730782    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df1da465-3751-4cf8-b8a8-e9faa35fd98f" path="/var/lib/kubelet/pods/df1da465-3751-4cf8-b8a8-e9faa35fd98f/volumes"
Sep 15 12:08:29.731239 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.731219    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb869d8e-73ba-4545-956a-e42501dd8ec9" path="/var/lib/kubelet/pods/fb869d8e-73ba-4545-956a-e42501dd8ec9/volumes"
Sep 15 12:08:29.746675 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.746651    2580 generic.go:356] "Generic (PLEG): container finished" podID="856006a0-813c-40e5-aca3-e6937e75a532" containerID="34d915be02f327e5169fe2ed67f4130b33c6f4ecb9c07b0772dc1b985eba3193" exitCode=0
Sep 15 12:08:29.746788 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.746707    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"
Sep 15 12:08:29.746788 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.746738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4" event={"ID":"856006a0-813c-40e5-aca3-e6937e75a532","Type":"ContainerDied","Data":"34d915be02f327e5169fe2ed67f4130b33c6f4ecb9c07b0772dc1b985eba3193"}
Sep 15 12:08:29.746788 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.746785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4" event={"ID":"856006a0-813c-40e5-aca3-e6937e75a532","Type":"ContainerDied","Data":"04f2f176a2ac3b199b8557c8670e94cfd4d8d2ea6978ce3191cda17efbb1f963"}
Sep 15 12:08:29.746981 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.746804    2580 scope.go:117] "RemoveContainer" containerID="34d915be02f327e5169fe2ed67f4130b33c6f4ecb9c07b0772dc1b985eba3193"
Sep 15 12:08:29.755901 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.755861    2580 scope.go:117] "RemoveContainer" containerID="34d915be02f327e5169fe2ed67f4130b33c6f4ecb9c07b0772dc1b985eba3193"
Sep 15 12:08:29.756167 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:08:29.756149    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d915be02f327e5169fe2ed67f4130b33c6f4ecb9c07b0772dc1b985eba3193\": container with ID starting with 34d915be02f327e5169fe2ed67f4130b33c6f4ecb9c07b0772dc1b985eba3193 not found: ID does not exist" containerID="34d915be02f327e5169fe2ed67f4130b33c6f4ecb9c07b0772dc1b985eba3193"
Sep 15 12:08:29.756227 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.756177    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d915be02f327e5169fe2ed67f4130b33c6f4ecb9c07b0772dc1b985eba3193"} err="failed to get container status \"34d915be02f327e5169fe2ed67f4130b33c6f4ecb9c07b0772dc1b985eba3193\": rpc error: code = NotFound desc = could not find container \"34d915be02f327e5169fe2ed67f4130b33c6f4ecb9c07b0772dc1b985eba3193\": container with ID starting with 34d915be02f327e5169fe2ed67f4130b33c6f4ecb9c07b0772dc1b985eba3193 not found: ID does not exist"
Sep 15 12:08:29.764233 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.764208    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"]
Sep 15 12:08:29.769026 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:29.769005    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/distinct-2-w4-ac76e5-kserve-7ccfc64495-n9pk4"]
Sep 15 12:08:31.730885 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:31.730849    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="856006a0-813c-40e5-aca3-e6937e75a532" path="/var/lib/kubelet/pods/856006a0-813c-40e5-aca3-e6937e75a532/volumes"
Sep 15 12:08:32.170257 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:32.170224    2580 scope.go:117] "RemoveContainer" containerID="0c1beb714879262ee86d0bf771f9302e69bc1790c376bdb19b43a29aa10f2426"
Sep 15 12:08:32.178492 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:32.178470    2580 scope.go:117] "RemoveContainer" containerID="0cd7de93fade0e5942d7d4e65c235da46856e239861dd7fef96fc7bbd5125e29"
Sep 15 12:08:58.597275 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:08:58.597240    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"]
Sep 15 12:09:13.616130 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:13.616087    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2" podUID="3e72aa51-3b91-433a-b376-cc146f292559" containerName="main" containerID="cri-o://415f765382a1a330df74d8cb8ae23a309665e67831c62dd032d74fcfb5e463ec" gracePeriod=45
Sep 15 12:09:13.865699 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:13.865674    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:09:13.982754 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:13.982714    2580 generic.go:356] "Generic (PLEG): container finished" podID="3e72aa51-3b91-433a-b376-cc146f292559" containerID="415f765382a1a330df74d8cb8ae23a309665e67831c62dd032d74fcfb5e463ec" exitCode=0
Sep 15 12:09:13.982967 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:13.982767    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"
Sep 15 12:09:13.982967 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:13.982784    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2" event={"ID":"3e72aa51-3b91-433a-b376-cc146f292559","Type":"ContainerDied","Data":"415f765382a1a330df74d8cb8ae23a309665e67831c62dd032d74fcfb5e463ec"}
Sep 15 12:09:13.982967 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:13.982816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2" event={"ID":"3e72aa51-3b91-433a-b376-cc146f292559","Type":"ContainerDied","Data":"530b8473c8a70e10a74723ea1e9444a230dcd2cf65290f344ef027d09d19c711"}
Sep 15 12:09:13.982967 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:13.982835    2580 scope.go:117] "RemoveContainer" containerID="415f765382a1a330df74d8cb8ae23a309665e67831c62dd032d74fcfb5e463ec"
Sep 15 12:09:13.991426 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:13.991389    2580 scope.go:117] "RemoveContainer" containerID="415f765382a1a330df74d8cb8ae23a309665e67831c62dd032d74fcfb5e463ec"
Sep 15 12:09:13.991688 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:09:13.991669    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415f765382a1a330df74d8cb8ae23a309665e67831c62dd032d74fcfb5e463ec\": container with ID starting with 415f765382a1a330df74d8cb8ae23a309665e67831c62dd032d74fcfb5e463ec not found: ID does not exist" containerID="415f765382a1a330df74d8cb8ae23a309665e67831c62dd032d74fcfb5e463ec"
Sep 15 12:09:13.991736 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:13.991697    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415f765382a1a330df74d8cb8ae23a309665e67831c62dd032d74fcfb5e463ec"} err="failed to get container status \"415f765382a1a330df74d8cb8ae23a309665e67831c62dd032d74fcfb5e463ec\": rpc error: code = NotFound desc = could not find container \"415f765382a1a330df74d8cb8ae23a309665e67831c62dd032d74fcfb5e463ec\": container with ID starting with 415f765382a1a330df74d8cb8ae23a309665e67831c62dd032d74fcfb5e463ec not found: ID does not exist"
Sep 15 12:09:13.999979 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:13.999961    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-model-cache\") pod \"3e72aa51-3b91-433a-b376-cc146f292559\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") "
Sep 15 12:09:14.000059 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.000048    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-home\") pod \"3e72aa51-3b91-433a-b376-cc146f292559\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") "
Sep 15 12:09:14.000128 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.000076    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e72aa51-3b91-433a-b376-cc146f292559-tls-certs\") pod \"3e72aa51-3b91-433a-b376-cc146f292559\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") "
Sep 15 12:09:14.000128 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.000109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-dshm\") pod \"3e72aa51-3b91-433a-b376-cc146f292559\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") "
Sep 15 12:09:14.000233 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.000134    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-tmp-dir\") pod \"3e72aa51-3b91-433a-b376-cc146f292559\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") "
Sep 15 12:09:14.000233 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.000182    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhcvq\" (UniqueName: \"kubernetes.io/projected/3e72aa51-3b91-433a-b376-cc146f292559-kube-api-access-rhcvq\") pod \"3e72aa51-3b91-433a-b376-cc146f292559\" (UID: \"3e72aa51-3b91-433a-b376-cc146f292559\") "
Sep 15 12:09:14.000327 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.000249    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e72aa51-3b91-433a-b376-cc146f292559" (UID: "3e72aa51-3b91-433a-b376-cc146f292559"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:09:14.000327 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.000287    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-home" (OuterVolumeSpecName: "home") pod "3e72aa51-3b91-433a-b376-cc146f292559" (UID: "3e72aa51-3b91-433a-b376-cc146f292559"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:09:14.000452 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.000422    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e72aa51-3b91-433a-b376-cc146f292559" (UID: "3e72aa51-3b91-433a-b376-cc146f292559"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:09:14.000565 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.000470    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:09:14.000565 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.000488    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:09:14.002331 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.002312    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e72aa51-3b91-433a-b376-cc146f292559-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e72aa51-3b91-433a-b376-cc146f292559" (UID: "3e72aa51-3b91-433a-b376-cc146f292559"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:09:14.002331 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.002326    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e72aa51-3b91-433a-b376-cc146f292559-kube-api-access-rhcvq" (OuterVolumeSpecName: "kube-api-access-rhcvq") pod "3e72aa51-3b91-433a-b376-cc146f292559" (UID: "3e72aa51-3b91-433a-b376-cc146f292559"). InnerVolumeSpecName "kube-api-access-rhcvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:09:14.002456 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.002380    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-dshm" (OuterVolumeSpecName: "dshm") pod "3e72aa51-3b91-433a-b376-cc146f292559" (UID: "3e72aa51-3b91-433a-b376-cc146f292559"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:09:14.101966 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.101933    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e72aa51-3b91-433a-b376-cc146f292559-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:09:14.101966 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.101963    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:09:14.101966 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.101972    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e72aa51-3b91-433a-b376-cc146f292559-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:09:14.102195 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.101982    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhcvq\" (UniqueName: \"kubernetes.io/projected/3e72aa51-3b91-433a-b376-cc146f292559-kube-api-access-rhcvq\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:09:14.305611 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.305580    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"]
Sep 15 12:09:14.310207 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:14.310181    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/auth-test-model-2b3936-kserve-5fcbb684ff-c9vj2"]
Sep 15 12:09:15.730933 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:15.730896    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e72aa51-3b91-433a-b376-cc146f292559" path="/var/lib/kubelet/pods/3e72aa51-3b91-433a-b376-cc146f292559/volumes"
Sep 15 12:09:26.867836 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.867794    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"]
Sep 15 12:09:26.868399 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868300    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e72aa51-3b91-433a-b376-cc146f292559" containerName="main"
Sep 15 12:09:26.868399 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868321    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e72aa51-3b91-433a-b376-cc146f292559" containerName="main"
Sep 15 12:09:26.868399 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868343    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ae2c82-2ef5-4ea4-8878-71c719d6868b" containerName="revoke-keys"
Sep 15 12:09:26.868399 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868352    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ae2c82-2ef5-4ea4-8878-71c719d6868b" containerName="revoke-keys"
Sep 15 12:09:26.868399 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868365    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df1da465-3751-4cf8-b8a8-e9faa35fd98f" containerName="main"
Sep 15 12:09:26.868399 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868373    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="df1da465-3751-4cf8-b8a8-e9faa35fd98f" containerName="main"
Sep 15 12:09:26.868399 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868383    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb869d8e-73ba-4545-956a-e42501dd8ec9" containerName="main"
Sep 15 12:09:26.868399 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868390    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb869d8e-73ba-4545-956a-e42501dd8ec9" containerName="main"
Sep 15 12:09:26.868810 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868414    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40091be4-fed1-4e6b-bded-82882eda3801" containerName="main"
Sep 15 12:09:26.868810 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868422    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40091be4-fed1-4e6b-bded-82882eda3801" containerName="main"
Sep 15 12:09:26.868810 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868435    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856006a0-813c-40e5-aca3-e6937e75a532" containerName="main"
Sep 15 12:09:26.868810 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868444    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="856006a0-813c-40e5-aca3-e6937e75a532" containerName="main"
Sep 15 12:09:26.868810 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868522    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e72aa51-3b91-433a-b376-cc146f292559" containerName="main"
Sep 15 12:09:26.868810 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868534    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2ae2c82-2ef5-4ea4-8878-71c719d6868b" containerName="revoke-keys"
Sep 15 12:09:26.868810 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868548    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb869d8e-73ba-4545-956a-e42501dd8ec9" containerName="main"
Sep 15 12:09:26.868810 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868557    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="df1da465-3751-4cf8-b8a8-e9faa35fd98f" containerName="main"
Sep 15 12:09:26.868810 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868565    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="856006a0-813c-40e5-aca3-e6937e75a532" containerName="main"
Sep 15 12:09:26.868810 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.868574    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40091be4-fed1-4e6b-bded-82882eda3801" containerName="main"
Sep 15 12:09:26.873178 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.873156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:26.877136 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.877112    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b3936\"/\"kube-root-ca.crt\""
Sep 15 12:09:26.877266 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.877112    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b3936\"/\"sub-test-model-2b3936-kserve-self-signed-certs\""
Sep 15 12:09:26.877266 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.877114    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b3936\"/\"openshift-service-ca.crt\""
Sep 15 12:09:26.877266 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.877114    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b3936\"/\"default-dockercfg-j2s5q\""
Sep 15 12:09:26.881126 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.881103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"]
Sep 15 12:09:26.919347 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.919316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-dshm\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:26.919471 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.919366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102799c9-82db-42a8-a4b1-7ef351617c88-tls-certs\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:26.919471 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.919419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-model-cache\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:26.919545 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.919491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-home\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:26.919589 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.919570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-tmp-dir\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:26.919624 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:26.919600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qncxm\" (UniqueName: \"kubernetes.io/projected/102799c9-82db-42a8-a4b1-7ef351617c88-kube-api-access-qncxm\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:27.020031 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:27.019995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-dshm\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:27.020031 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:27.020034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102799c9-82db-42a8-a4b1-7ef351617c88-tls-certs\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:27.020241 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:27.020057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-model-cache\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:27.020241 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:27.020096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-home\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:27.020241 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:27.020131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-tmp-dir\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:27.020241 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:27.020158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qncxm\" (UniqueName: \"kubernetes.io/projected/102799c9-82db-42a8-a4b1-7ef351617c88-kube-api-access-qncxm\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:27.020514 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:27.020485    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-model-cache\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:27.020641 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:27.020518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-home\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:27.020641 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:27.020563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-tmp-dir\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:27.022361 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:27.022334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-dshm\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:27.022603 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:27.022584    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102799c9-82db-42a8-a4b1-7ef351617c88-tls-certs\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:27.029129 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:27.029109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qncxm\" (UniqueName: \"kubernetes.io/projected/102799c9-82db-42a8-a4b1-7ef351617c88-kube-api-access-qncxm\") pod \"sub-test-model-2b3936-kserve-66997cdfb9-9n4cp\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:27.183637 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:27.183609    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:27.460659 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:27.460629    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"]
Sep 15 12:09:27.461921 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:09:27.461887    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod102799c9_82db_42a8_a4b1_7ef351617c88.slice/crio-af81f77a85f84821e8f2d668ca2446304f57604e4f8ad5b28d07d971cb67c74c WatchSource:0}: Error finding container af81f77a85f84821e8f2d668ca2446304f57604e4f8ad5b28d07d971cb67c74c: Status 404 returned error can't find the container with id af81f77a85f84821e8f2d668ca2446304f57604e4f8ad5b28d07d971cb67c74c
Sep 15 12:09:28.063452 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:28.063406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp" event={"ID":"102799c9-82db-42a8-a4b1-7ef351617c88","Type":"ContainerStarted","Data":"4835894ea626b84ad14527ef19a45ea3f42d3604b6f3045af783df010b2c6816"}
Sep 15 12:09:28.063824 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:28.063462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp" event={"ID":"102799c9-82db-42a8-a4b1-7ef351617c88","Type":"ContainerStarted","Data":"af81f77a85f84821e8f2d668ca2446304f57604e4f8ad5b28d07d971cb67c74c"}
Sep 15 12:09:28.084481 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:28.084427    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp" podStartSLOduration=2.084412473 podStartE2EDuration="2.084412473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:09:28.083258651 +0000 UTC m=+1856.863830040" watchObservedRunningTime="2026-09-15 12:09:28.084412473 +0000 UTC m=+1856.864983861"
Sep 15 12:09:37.184516 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:37.184480    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:37.184516 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:37.184526    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:37.197333 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:37.197309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:09:38.126335 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:09:38.126307    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:10:02.278491 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.278411    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"]
Sep 15 12:10:02.281778 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.281760    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.285241 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.285217    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-32bbbb\"/\"kube-root-ca.crt\""
Sep 15 12:10:02.285397 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.285378    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-32bbbb\"/\"default-dockercfg-l95jw\""
Sep 15 12:10:02.285472 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.285446    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-32bbbb\"/\"openshift-service-ca.crt\""
Sep 15 12:10:02.285536 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.285519    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-32bbbb\"/\"sub-test-model-32bbbb-kserve-self-signed-certs\""
Sep 15 12:10:02.292270 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.292246    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"]
Sep 15 12:10:02.427394 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.427353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-tmp-dir\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.427581 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.427422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v69m\" (UniqueName: \"kubernetes.io/projected/10c65e32-3843-4b8c-993c-a9a2bb962631-kube-api-access-7v69m\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.427581 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.427460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-model-cache\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.427581 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.427486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c65e32-3843-4b8c-993c-a9a2bb962631-tls-certs\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.427699 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.427575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-home\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.427699 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.427628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-dshm\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.529015 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.528925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-dshm\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.529015 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.528969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-tmp-dir\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.529250 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.529018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v69m\" (UniqueName: \"kubernetes.io/projected/10c65e32-3843-4b8c-993c-a9a2bb962631-kube-api-access-7v69m\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.529250 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.529040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-model-cache\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.529250 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.529072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c65e32-3843-4b8c-993c-a9a2bb962631-tls-certs\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.529250 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.529128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-home\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.529455 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.529423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-model-cache\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.529510 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.529451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-tmp-dir\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.529510 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.529470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-home\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.531320 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.531295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-dshm\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.531509 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.531487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c65e32-3843-4b8c-993c-a9a2bb962631-tls-certs\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.539422 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.539400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v69m\" (UniqueName: \"kubernetes.io/projected/10c65e32-3843-4b8c-993c-a9a2bb962631-kube-api-access-7v69m\") pod \"sub-test-model-32bbbb-kserve-75865647df-whwg4\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.591437 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.591408    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:02.867097 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:02.867066    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"]
Sep 15 12:10:02.868539 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:10:02.868512    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10c65e32_3843_4b8c_993c_a9a2bb962631.slice/crio-ff4191a6c068fc95f28577747adb54bf4283e5622471456e252ea15ae0e5335a WatchSource:0}: Error finding container ff4191a6c068fc95f28577747adb54bf4283e5622471456e252ea15ae0e5335a: Status 404 returned error can't find the container with id ff4191a6c068fc95f28577747adb54bf4283e5622471456e252ea15ae0e5335a
Sep 15 12:10:03.245859 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:03.245815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4" event={"ID":"10c65e32-3843-4b8c-993c-a9a2bb962631","Type":"ContainerStarted","Data":"4ea06b3511b01a9173bb48cb52024215dac6f1b39ab7593c29030b0a12d31eb0"}
Sep 15 12:10:03.245859 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:03.245865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4" event={"ID":"10c65e32-3843-4b8c-993c-a9a2bb962631","Type":"ContainerStarted","Data":"ff4191a6c068fc95f28577747adb54bf4283e5622471456e252ea15ae0e5335a"}
Sep 15 12:10:03.266270 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:03.266222    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4" podStartSLOduration=1.266208673 podStartE2EDuration="1.266208673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:03.264910883 +0000 UTC m=+1892.045482270" watchObservedRunningTime="2026-09-15 12:10:03.266208673 +0000 UTC m=+1892.046780061"
Sep 15 12:10:12.592349 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:12.592313    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:12.592743 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:12.592387    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:12.604935 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:12.604897    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:13.314118 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:13.314092    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:10:46.444469 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:46.444435    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"]
Sep 15 12:10:47.381259 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:47.381217    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"]
Sep 15 12:10:58.293643 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.293609    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"]
Sep 15 12:10:58.297306 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.297290    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.299452 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.299426    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2b3936\"/\"rate-test-model-2b3936-kserve-self-signed-certs\""
Sep 15 12:10:58.308511 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.308488    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"]
Sep 15 12:10:58.420628 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.420599    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-tmp-dir\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.420802 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.420660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-dshm\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.420802 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.420727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-home\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.420802 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.420757    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-model-cache\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.420802 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.420786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/263c9b19-494f-4404-a6aa-a26714006ab9-tls-certs\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.420994 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.420812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjfjr\" (UniqueName: \"kubernetes.io/projected/263c9b19-494f-4404-a6aa-a26714006ab9-kube-api-access-kjfjr\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.522090 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.522061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-dshm\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.522261 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.522097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-home\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.522261 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.522123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-model-cache\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.522261 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.522155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/263c9b19-494f-4404-a6aa-a26714006ab9-tls-certs\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.522261 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.522176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjfjr\" (UniqueName: \"kubernetes.io/projected/263c9b19-494f-4404-a6aa-a26714006ab9-kube-api-access-kjfjr\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.522261 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.522210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-tmp-dir\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.522591 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.522564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-home\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.522670 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.522644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-tmp-dir\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.522720 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.522700    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-model-cache\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.524425 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.524402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-dshm\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.524678 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.524659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/263c9b19-494f-4404-a6aa-a26714006ab9-tls-certs\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.531914 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.531867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjfjr\" (UniqueName: \"kubernetes.io/projected/263c9b19-494f-4404-a6aa-a26714006ab9-kube-api-access-kjfjr\") pod \"rate-test-model-2b3936-kserve-9d4b75847-sqt26\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") " pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.606010 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.605944    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:10:58.882530 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:58.882492    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"]
Sep 15 12:10:58.883838 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:10:58.883816    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod263c9b19_494f_4404_a6aa_a26714006ab9.slice/crio-38fc6d004ceb68b66f298d212ea0dc5d504b1984d5144ff43d19396c018cc677 WatchSource:0}: Error finding container 38fc6d004ceb68b66f298d212ea0dc5d504b1984d5144ff43d19396c018cc677: Status 404 returned error can't find the container with id 38fc6d004ceb68b66f298d212ea0dc5d504b1984d5144ff43d19396c018cc677
Sep 15 12:10:59.560678 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:59.560641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26" event={"ID":"263c9b19-494f-4404-a6aa-a26714006ab9","Type":"ContainerStarted","Data":"91c95fbd2a9e25c56afbc5762d992c533295254c64740229ed732c82f3b865c3"}
Sep 15 12:10:59.561064 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:59.560684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26" event={"ID":"263c9b19-494f-4404-a6aa-a26714006ab9","Type":"ContainerStarted","Data":"38fc6d004ceb68b66f298d212ea0dc5d504b1984d5144ff43d19396c018cc677"}
Sep 15 12:10:59.582136 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:10:59.582088    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26" podStartSLOduration=1.5820723829999999 podStartE2EDuration="1.582072383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:59.580090917 +0000 UTC m=+1948.360662305" watchObservedRunningTime="2026-09-15 12:10:59.582072383 +0000 UTC m=+1948.362643820"
Sep 15 12:11:01.463536 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.463493    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp" podUID="102799c9-82db-42a8-a4b1-7ef351617c88" containerName="main" containerID="cri-o://4835894ea626b84ad14527ef19a45ea3f42d3604b6f3045af783df010b2c6816" gracePeriod=45
Sep 15 12:11:01.713865 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.713809    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:11:01.849808 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.849768    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-model-cache\") pod \"102799c9-82db-42a8-a4b1-7ef351617c88\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") "
Sep 15 12:11:01.850001 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.849824    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-home\") pod \"102799c9-82db-42a8-a4b1-7ef351617c88\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") "
Sep 15 12:11:01.850001 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.849852    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-tmp-dir\") pod \"102799c9-82db-42a8-a4b1-7ef351617c88\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") "
Sep 15 12:11:01.850001 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.849935    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-dshm\") pod \"102799c9-82db-42a8-a4b1-7ef351617c88\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") "
Sep 15 12:11:01.850001 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.849957    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102799c9-82db-42a8-a4b1-7ef351617c88-tls-certs\") pod \"102799c9-82db-42a8-a4b1-7ef351617c88\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") "
Sep 15 12:11:01.850236 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.850028    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qncxm\" (UniqueName: \"kubernetes.io/projected/102799c9-82db-42a8-a4b1-7ef351617c88-kube-api-access-qncxm\") pod \"102799c9-82db-42a8-a4b1-7ef351617c88\" (UID: \"102799c9-82db-42a8-a4b1-7ef351617c88\") "
Sep 15 12:11:01.850236 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.850153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-model-cache" (OuterVolumeSpecName: "model-cache") pod "102799c9-82db-42a8-a4b1-7ef351617c88" (UID: "102799c9-82db-42a8-a4b1-7ef351617c88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:11:01.850236 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.850160    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-home" (OuterVolumeSpecName: "home") pod "102799c9-82db-42a8-a4b1-7ef351617c88" (UID: "102799c9-82db-42a8-a4b1-7ef351617c88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:11:01.850406 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.850247    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "102799c9-82db-42a8-a4b1-7ef351617c88" (UID: "102799c9-82db-42a8-a4b1-7ef351617c88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:11:01.850478 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.850459    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:11:01.850543 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.850483    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:11:01.850543 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.850496    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:11:01.852186 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.852163    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102799c9-82db-42a8-a4b1-7ef351617c88-kube-api-access-qncxm" (OuterVolumeSpecName: "kube-api-access-qncxm") pod "102799c9-82db-42a8-a4b1-7ef351617c88" (UID: "102799c9-82db-42a8-a4b1-7ef351617c88"). InnerVolumeSpecName "kube-api-access-qncxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:11:01.852519 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.852494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-dshm" (OuterVolumeSpecName: "dshm") pod "102799c9-82db-42a8-a4b1-7ef351617c88" (UID: "102799c9-82db-42a8-a4b1-7ef351617c88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:11:01.852519 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.852507    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102799c9-82db-42a8-a4b1-7ef351617c88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "102799c9-82db-42a8-a4b1-7ef351617c88" (UID: "102799c9-82db-42a8-a4b1-7ef351617c88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:11:01.951573 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.951530    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102799c9-82db-42a8-a4b1-7ef351617c88-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:11:01.951573 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.951570    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102799c9-82db-42a8-a4b1-7ef351617c88-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:11:01.951573 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:01.951581    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qncxm\" (UniqueName: \"kubernetes.io/projected/102799c9-82db-42a8-a4b1-7ef351617c88-kube-api-access-qncxm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:11:02.399887 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.399814    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4" podUID="10c65e32-3843-4b8c-993c-a9a2bb962631" containerName="main" containerID="cri-o://4ea06b3511b01a9173bb48cb52024215dac6f1b39ab7593c29030b0a12d31eb0" gracePeriod=45
Sep 15 12:11:02.583471 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.583433    2580 generic.go:356] "Generic (PLEG): container finished" podID="102799c9-82db-42a8-a4b1-7ef351617c88" containerID="4835894ea626b84ad14527ef19a45ea3f42d3604b6f3045af783df010b2c6816" exitCode=0
Sep 15 12:11:02.583840 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.583472    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"
Sep 15 12:11:02.583840 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.583510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp" event={"ID":"102799c9-82db-42a8-a4b1-7ef351617c88","Type":"ContainerDied","Data":"4835894ea626b84ad14527ef19a45ea3f42d3604b6f3045af783df010b2c6816"}
Sep 15 12:11:02.583840 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.583561    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp" event={"ID":"102799c9-82db-42a8-a4b1-7ef351617c88","Type":"ContainerDied","Data":"af81f77a85f84821e8f2d668ca2446304f57604e4f8ad5b28d07d971cb67c74c"}
Sep 15 12:11:02.584325 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.583586    2580 scope.go:117] "RemoveContainer" containerID="4835894ea626b84ad14527ef19a45ea3f42d3604b6f3045af783df010b2c6816"
Sep 15 12:11:02.585716 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.585666    2580 generic.go:356] "Generic (PLEG): container finished" podID="10c65e32-3843-4b8c-993c-a9a2bb962631" containerID="4ea06b3511b01a9173bb48cb52024215dac6f1b39ab7593c29030b0a12d31eb0" exitCode=0
Sep 15 12:11:02.586099 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.585713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4" event={"ID":"10c65e32-3843-4b8c-993c-a9a2bb962631","Type":"ContainerDied","Data":"4ea06b3511b01a9173bb48cb52024215dac6f1b39ab7593c29030b0a12d31eb0"}
Sep 15 12:11:02.595566 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.595544    2580 scope.go:117] "RemoveContainer" containerID="4835894ea626b84ad14527ef19a45ea3f42d3604b6f3045af783df010b2c6816"
Sep 15 12:11:02.595928 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:11:02.595901    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4835894ea626b84ad14527ef19a45ea3f42d3604b6f3045af783df010b2c6816\": container with ID starting with 4835894ea626b84ad14527ef19a45ea3f42d3604b6f3045af783df010b2c6816 not found: ID does not exist" containerID="4835894ea626b84ad14527ef19a45ea3f42d3604b6f3045af783df010b2c6816"
Sep 15 12:11:02.596029 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.595939    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4835894ea626b84ad14527ef19a45ea3f42d3604b6f3045af783df010b2c6816"} err="failed to get container status \"4835894ea626b84ad14527ef19a45ea3f42d3604b6f3045af783df010b2c6816\": rpc error: code = NotFound desc = could not find container \"4835894ea626b84ad14527ef19a45ea3f42d3604b6f3045af783df010b2c6816\": container with ID starting with 4835894ea626b84ad14527ef19a45ea3f42d3604b6f3045af783df010b2c6816 not found: ID does not exist"
Sep 15 12:11:02.608557 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.608528    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"]
Sep 15 12:11:02.618681 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.618656    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/sub-test-model-2b3936-kserve-66997cdfb9-9n4cp"]
Sep 15 12:11:02.643727 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.643701    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:11:02.760014 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.759979    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-model-cache\") pod \"10c65e32-3843-4b8c-993c-a9a2bb962631\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") "
Sep 15 12:11:02.760014 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.760020    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-tmp-dir\") pod \"10c65e32-3843-4b8c-993c-a9a2bb962631\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") "
Sep 15 12:11:02.760231 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.760054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-home\") pod \"10c65e32-3843-4b8c-993c-a9a2bb962631\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") "
Sep 15 12:11:02.760231 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.760089    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-dshm\") pod \"10c65e32-3843-4b8c-993c-a9a2bb962631\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") "
Sep 15 12:11:02.760231 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.760129    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c65e32-3843-4b8c-993c-a9a2bb962631-tls-certs\") pod \"10c65e32-3843-4b8c-993c-a9a2bb962631\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") "
Sep 15 12:11:02.760231 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.760160    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v69m\" (UniqueName: \"kubernetes.io/projected/10c65e32-3843-4b8c-993c-a9a2bb962631-kube-api-access-7v69m\") pod \"10c65e32-3843-4b8c-993c-a9a2bb962631\" (UID: \"10c65e32-3843-4b8c-993c-a9a2bb962631\") "
Sep 15 12:11:02.760445 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.760310    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-model-cache" (OuterVolumeSpecName: "model-cache") pod "10c65e32-3843-4b8c-993c-a9a2bb962631" (UID: "10c65e32-3843-4b8c-993c-a9a2bb962631"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:11:02.760445 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.760320    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10c65e32-3843-4b8c-993c-a9a2bb962631" (UID: "10c65e32-3843-4b8c-993c-a9a2bb962631"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:11:02.760445 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.760415    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-home" (OuterVolumeSpecName: "home") pod "10c65e32-3843-4b8c-993c-a9a2bb962631" (UID: "10c65e32-3843-4b8c-993c-a9a2bb962631"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:11:02.760597 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.760495    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:11:02.760597 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.760512    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:11:02.760597 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.760525    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:11:02.762468 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.762439    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c65e32-3843-4b8c-993c-a9a2bb962631-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10c65e32-3843-4b8c-993c-a9a2bb962631" (UID: "10c65e32-3843-4b8c-993c-a9a2bb962631"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:11:02.762566 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.762468    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-dshm" (OuterVolumeSpecName: "dshm") pod "10c65e32-3843-4b8c-993c-a9a2bb962631" (UID: "10c65e32-3843-4b8c-993c-a9a2bb962631"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:11:02.762566 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.762533    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c65e32-3843-4b8c-993c-a9a2bb962631-kube-api-access-7v69m" (OuterVolumeSpecName: "kube-api-access-7v69m") pod "10c65e32-3843-4b8c-993c-a9a2bb962631" (UID: "10c65e32-3843-4b8c-993c-a9a2bb962631"). InnerVolumeSpecName "kube-api-access-7v69m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:11:02.861919 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.861862    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v69m\" (UniqueName: \"kubernetes.io/projected/10c65e32-3843-4b8c-993c-a9a2bb962631-kube-api-access-7v69m\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:11:02.861919 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.861914    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c65e32-3843-4b8c-993c-a9a2bb962631-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:11:02.861919 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:02.861923    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c65e32-3843-4b8c-993c-a9a2bb962631-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:11:03.593765 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:03.593734    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"
Sep 15 12:11:03.594305 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:03.593745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4" event={"ID":"10c65e32-3843-4b8c-993c-a9a2bb962631","Type":"ContainerDied","Data":"ff4191a6c068fc95f28577747adb54bf4283e5622471456e252ea15ae0e5335a"}
Sep 15 12:11:03.594305 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:03.593857    2580 scope.go:117] "RemoveContainer" containerID="4ea06b3511b01a9173bb48cb52024215dac6f1b39ab7593c29030b0a12d31eb0"
Sep 15 12:11:03.617244 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:03.617215    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"]
Sep 15 12:11:03.621281 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:03.621258    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32bbbb/sub-test-model-32bbbb-kserve-75865647df-whwg4"]
Sep 15 12:11:03.731106 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:03.731072    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="102799c9-82db-42a8-a4b1-7ef351617c88" path="/var/lib/kubelet/pods/102799c9-82db-42a8-a4b1-7ef351617c88/volumes"
Sep 15 12:11:03.731491 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:03.731477    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c65e32-3843-4b8c-993c-a9a2bb962631" path="/var/lib/kubelet/pods/10c65e32-3843-4b8c-993c-a9a2bb962631/volumes"
Sep 15 12:11:08.606533 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:08.606494    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:11:08.606971 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:08.606730    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:11:08.619297 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:08.619274    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:11:08.632383 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:08.632357    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:11:17.572961 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.572921    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"]
Sep 15 12:11:17.573476 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.573395    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c65e32-3843-4b8c-993c-a9a2bb962631" containerName="main"
Sep 15 12:11:17.573476 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.573409    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c65e32-3843-4b8c-993c-a9a2bb962631" containerName="main"
Sep 15 12:11:17.573476 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.573448    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="102799c9-82db-42a8-a4b1-7ef351617c88" containerName="main"
Sep 15 12:11:17.573476 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.573457    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="102799c9-82db-42a8-a4b1-7ef351617c88" containerName="main"
Sep 15 12:11:17.573716 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.573538    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c65e32-3843-4b8c-993c-a9a2bb962631" containerName="main"
Sep 15 12:11:17.573716 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.573552    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="102799c9-82db-42a8-a4b1-7ef351617c88" containerName="main"
Sep 15 12:11:17.576920 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.576898    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.580374 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.580353    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-32bbbb\"/\"default-dockercfg-l95jw\""
Sep 15 12:11:17.580519 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.580366    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-32bbbb\"/\"rate-test-model-32bbbb-kserve-self-signed-certs\""
Sep 15 12:11:17.580519 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.580387    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-32bbbb\"/\"kube-root-ca.crt\""
Sep 15 12:11:17.580519 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.580367    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-32bbbb\"/\"openshift-service-ca.crt\""
Sep 15 12:11:17.587400 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.587375    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"]
Sep 15 12:11:17.690548 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.690515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8qcm\" (UniqueName: \"kubernetes.io/projected/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-kube-api-access-j8qcm\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.690755 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.690565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-tls-certs\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.690755 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.690639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-dshm\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.690755 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.690673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-tmp-dir\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.690755 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.690702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-home\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.690952 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.690762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-model-cache\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.791142 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.791111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-tmp-dir\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.791142 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.791143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-home\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.791391 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.791189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-model-cache\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.791391 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.791260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8qcm\" (UniqueName: \"kubernetes.io/projected/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-kube-api-access-j8qcm\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.791391 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.791328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-tls-certs\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.791391 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.791377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-dshm\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.791613 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.791533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-tmp-dir\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.791613 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.791570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-model-cache\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.791743 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.791721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-home\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.793670 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.793642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-dshm\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.793981 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.793959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-tls-certs\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.808305 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.801380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8qcm\" (UniqueName: \"kubernetes.io/projected/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-kube-api-access-j8qcm\") pod \"rate-test-model-32bbbb-kserve-8b669f79-r2rkw\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") " pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:17.887993 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:17.887918    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:18.163373 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:18.163342    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"]
Sep 15 12:11:18.164992 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:11:18.164965    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab4a4867_caec_44de_bfb4_e9af30e6a7e1.slice/crio-2fa88715b65351b441e1bb82181449a380fa02b2f42aa65dc8186437b5d864b6 WatchSource:0}: Error finding container 2fa88715b65351b441e1bb82181449a380fa02b2f42aa65dc8186437b5d864b6: Status 404 returned error can't find the container with id 2fa88715b65351b441e1bb82181449a380fa02b2f42aa65dc8186437b5d864b6
Sep 15 12:11:18.682280 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:18.682244    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw" event={"ID":"ab4a4867-caec-44de-bfb4-e9af30e6a7e1","Type":"ContainerStarted","Data":"fbcf93879a1762c3d1e449019abfc7118b578e4dc5b088099ba4d90d94a49dbc"}
Sep 15 12:11:18.682685 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:18.682289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw" event={"ID":"ab4a4867-caec-44de-bfb4-e9af30e6a7e1","Type":"ContainerStarted","Data":"2fa88715b65351b441e1bb82181449a380fa02b2f42aa65dc8186437b5d864b6"}
Sep 15 12:11:18.701728 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:18.701684    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw" podStartSLOduration=1.701669872 podStartE2EDuration="1.701669872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:11:18.699997284 +0000 UTC m=+1967.480568672" watchObservedRunningTime="2026-09-15 12:11:18.701669872 +0000 UTC m=+1967.482241260"
Sep 15 12:11:27.888422 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:27.888335    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:27.888422 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:27.888385    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:27.900816 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:27.900793    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:28.747553 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:28.747519    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:11:32.239883 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:11:32.239850    2580 scope.go:117] "RemoveContainer" containerID="a886c78f782be8dbb4287b43ca259992ad47a3c9011a22d92d2d5437654a7c74"
Sep 15 12:12:06.421494 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:06.421453    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"]
Sep 15 12:12:07.377883 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:07.377844    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"]
Sep 15 12:12:08.146576 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.146533    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"]
Sep 15 12:12:08.150249 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.150226    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.152708 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.152624    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7139fb-openshift-default-dockercfg-g652j\""
Sep 15 12:12:08.167251 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.167229    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"]
Sep 15 12:12:08.224134 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.224111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-workload-socket\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.224282 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.224161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-token\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.224282 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.224194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t4pd\" (UniqueName: \"kubernetes.io/projected/223aa62e-8850-4abe-9961-4ddbb8df9131-kube-api-access-6t4pd\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.224282 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.224230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-podinfo\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.224282 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.224263    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-data\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.224452 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.224288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-credential-socket\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.224452 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.224305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-workload-certs\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.224452 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.224334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-envoy\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.224452 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.224357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223aa62e-8850-4abe-9961-4ddbb8df9131-istiod-ca-cert\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.325573 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.325534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-credential-socket\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.325573 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.325577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-workload-certs\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.325814 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.325617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-envoy\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.325814 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.325639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223aa62e-8850-4abe-9961-4ddbb8df9131-istiod-ca-cert\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.325814 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.325674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-workload-socket\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.325814 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.325731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-token\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.325814 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.325772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t4pd\" (UniqueName: \"kubernetes.io/projected/223aa62e-8850-4abe-9961-4ddbb8df9131-kube-api-access-6t4pd\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.325814 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.325801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-podinfo\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.326254 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.325855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-data\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.326254 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.326052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-workload-certs\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.326254 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.326096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-workload-socket\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.326254 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.326222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-data\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.326410 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.326289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-credential-socket\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.326489 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.326463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223aa62e-8850-4abe-9961-4ddbb8df9131-istiod-ca-cert\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.328086 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.328061    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-envoy\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.328362 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.328346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-podinfo\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.334183 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.334156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-token\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.334340 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.334322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t4pd\" (UniqueName: \"kubernetes.io/projected/223aa62e-8850-4abe-9961-4ddbb8df9131-kube-api-access-6t4pd\") pod \"e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") " pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.469200 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.469170    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:08.750068 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.750044    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"]
Sep 15 12:12:08.752054 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:12:08.752027    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223aa62e_8850_4abe_9961_4ddbb8df9131.slice/crio-40e162f9f61d4ea9c950ca94a094665539cdc83d18546245ba0bb2e02c23f816 WatchSource:0}: Error finding container 40e162f9f61d4ea9c950ca94a094665539cdc83d18546245ba0bb2e02c23f816: Status 404 returned error can't find the container with id 40e162f9f61d4ea9c950ca94a094665539cdc83d18546245ba0bb2e02c23f816
Sep 15 12:12:08.754378 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.754344    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:12:08.754469 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.754409    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:12:08.754469 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.754458    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:12:08.955105 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.955058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp" event={"ID":"223aa62e-8850-4abe-9961-4ddbb8df9131","Type":"ContainerStarted","Data":"7d65bc924864dfa4639221b37f2d1369c85608416ecdf816bfbfe7d7c4955a93"}
Sep 15 12:12:08.955276 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.955137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp" event={"ID":"223aa62e-8850-4abe-9961-4ddbb8df9131","Type":"ContainerStarted","Data":"40e162f9f61d4ea9c950ca94a094665539cdc83d18546245ba0bb2e02c23f816"}
Sep 15 12:12:08.980308 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:08.980252    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp" podStartSLOduration=0.98023541 podStartE2EDuration="980.23541ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:12:08.977691384 +0000 UTC m=+2017.758262773" watchObservedRunningTime="2026-09-15 12:12:08.98023541 +0000 UTC m=+2017.760806797"
Sep 15 12:12:09.469799 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:09.469765    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:09.474584 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:09.474561    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:09.960409 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:09.960379    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:09.961267 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:09.961247    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:21.439931 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.439856    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26" podUID="263c9b19-494f-4404-a6aa-a26714006ab9" containerName="main" containerID="cri-o://91c95fbd2a9e25c56afbc5762d992c533295254c64740229ed732c82f3b865c3" gracePeriod=45
Sep 15 12:12:21.717282 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.717257    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:12:21.873464 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.873410    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/263c9b19-494f-4404-a6aa-a26714006ab9-tls-certs\") pod \"263c9b19-494f-4404-a6aa-a26714006ab9\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") "
Sep 15 12:12:21.873628 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.873475    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-dshm\") pod \"263c9b19-494f-4404-a6aa-a26714006ab9\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") "
Sep 15 12:12:21.873628 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.873518    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-home\") pod \"263c9b19-494f-4404-a6aa-a26714006ab9\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") "
Sep 15 12:12:21.873628 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.873618    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjfjr\" (UniqueName: \"kubernetes.io/projected/263c9b19-494f-4404-a6aa-a26714006ab9-kube-api-access-kjfjr\") pod \"263c9b19-494f-4404-a6aa-a26714006ab9\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") "
Sep 15 12:12:21.873811 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.873648    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-tmp-dir\") pod \"263c9b19-494f-4404-a6aa-a26714006ab9\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") "
Sep 15 12:12:21.873811 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.873730    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-model-cache\") pod \"263c9b19-494f-4404-a6aa-a26714006ab9\" (UID: \"263c9b19-494f-4404-a6aa-a26714006ab9\") "
Sep 15 12:12:21.873974 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.873815    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-home" (OuterVolumeSpecName: "home") pod "263c9b19-494f-4404-a6aa-a26714006ab9" (UID: "263c9b19-494f-4404-a6aa-a26714006ab9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:12:21.873974 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.873920    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "263c9b19-494f-4404-a6aa-a26714006ab9" (UID: "263c9b19-494f-4404-a6aa-a26714006ab9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:12:21.874099 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.874030    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-model-cache" (OuterVolumeSpecName: "model-cache") pod "263c9b19-494f-4404-a6aa-a26714006ab9" (UID: "263c9b19-494f-4404-a6aa-a26714006ab9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:12:21.874099 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.874088    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:12:21.874206 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.874104    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:12:21.874206 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.874112    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:12:21.875638 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.875603    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-dshm" (OuterVolumeSpecName: "dshm") pod "263c9b19-494f-4404-a6aa-a26714006ab9" (UID: "263c9b19-494f-4404-a6aa-a26714006ab9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:12:21.876112 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.876086    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/263c9b19-494f-4404-a6aa-a26714006ab9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "263c9b19-494f-4404-a6aa-a26714006ab9" (UID: "263c9b19-494f-4404-a6aa-a26714006ab9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:12:21.876112 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.876103    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263c9b19-494f-4404-a6aa-a26714006ab9-kube-api-access-kjfjr" (OuterVolumeSpecName: "kube-api-access-kjfjr") pod "263c9b19-494f-4404-a6aa-a26714006ab9" (UID: "263c9b19-494f-4404-a6aa-a26714006ab9"). InnerVolumeSpecName "kube-api-access-kjfjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:12:21.975014 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.974922    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjfjr\" (UniqueName: \"kubernetes.io/projected/263c9b19-494f-4404-a6aa-a26714006ab9-kube-api-access-kjfjr\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:12:21.975014 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.974956    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/263c9b19-494f-4404-a6aa-a26714006ab9-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:12:21.975014 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:21.974967    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/263c9b19-494f-4404-a6aa-a26714006ab9-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:12:22.029135 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.029101    2580 generic.go:356] "Generic (PLEG): container finished" podID="263c9b19-494f-4404-a6aa-a26714006ab9" containerID="91c95fbd2a9e25c56afbc5762d992c533295254c64740229ed732c82f3b865c3" exitCode=0
Sep 15 12:12:22.029308 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.029143    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"
Sep 15 12:12:22.029308 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.029184    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26" event={"ID":"263c9b19-494f-4404-a6aa-a26714006ab9","Type":"ContainerDied","Data":"91c95fbd2a9e25c56afbc5762d992c533295254c64740229ed732c82f3b865c3"}
Sep 15 12:12:22.029308 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.029231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26" event={"ID":"263c9b19-494f-4404-a6aa-a26714006ab9","Type":"ContainerDied","Data":"38fc6d004ceb68b66f298d212ea0dc5d504b1984d5144ff43d19396c018cc677"}
Sep 15 12:12:22.029308 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.029249    2580 scope.go:117] "RemoveContainer" containerID="91c95fbd2a9e25c56afbc5762d992c533295254c64740229ed732c82f3b865c3"
Sep 15 12:12:22.039175 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.039158    2580 scope.go:117] "RemoveContainer" containerID="91c95fbd2a9e25c56afbc5762d992c533295254c64740229ed732c82f3b865c3"
Sep 15 12:12:22.039444 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:12:22.039425    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c95fbd2a9e25c56afbc5762d992c533295254c64740229ed732c82f3b865c3\": container with ID starting with 91c95fbd2a9e25c56afbc5762d992c533295254c64740229ed732c82f3b865c3 not found: ID does not exist" containerID="91c95fbd2a9e25c56afbc5762d992c533295254c64740229ed732c82f3b865c3"
Sep 15 12:12:22.039489 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.039453    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c95fbd2a9e25c56afbc5762d992c533295254c64740229ed732c82f3b865c3"} err="failed to get container status \"91c95fbd2a9e25c56afbc5762d992c533295254c64740229ed732c82f3b865c3\": rpc error: code = NotFound desc = could not find container \"91c95fbd2a9e25c56afbc5762d992c533295254c64740229ed732c82f3b865c3\": container with ID starting with 91c95fbd2a9e25c56afbc5762d992c533295254c64740229ed732c82f3b865c3 not found: ID does not exist"
Sep 15 12:12:22.067368 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.067321    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"]
Sep 15 12:12:22.067574 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.067387    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2b3936/rate-test-model-2b3936-kserve-9d4b75847-sqt26"]
Sep 15 12:12:22.396375 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.396267    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw" podUID="ab4a4867-caec-44de-bfb4-e9af30e6a7e1" containerName="main" containerID="cri-o://fbcf93879a1762c3d1e449019abfc7118b578e4dc5b088099ba4d90d94a49dbc" gracePeriod=45
Sep 15 12:12:22.684653 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.684624    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:12:22.782200 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.782165    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-home\") pod \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") "
Sep 15 12:12:22.782374 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.782243    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8qcm\" (UniqueName: \"kubernetes.io/projected/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-kube-api-access-j8qcm\") pod \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") "
Sep 15 12:12:22.782374 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.782282    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-tmp-dir\") pod \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") "
Sep 15 12:12:22.782374 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.782314    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-dshm\") pod \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") "
Sep 15 12:12:22.782548 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.782457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-home" (OuterVolumeSpecName: "home") pod "ab4a4867-caec-44de-bfb4-e9af30e6a7e1" (UID: "ab4a4867-caec-44de-bfb4-e9af30e6a7e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:12:22.782548 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.782515    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-tls-certs\") pod \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") "
Sep 15 12:12:22.782651 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.782601    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-model-cache\") pod \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\" (UID: \"ab4a4867-caec-44de-bfb4-e9af30e6a7e1\") "
Sep 15 12:12:22.782651 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.782607    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab4a4867-caec-44de-bfb4-e9af30e6a7e1" (UID: "ab4a4867-caec-44de-bfb4-e9af30e6a7e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:12:22.782864 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.782842    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab4a4867-caec-44de-bfb4-e9af30e6a7e1" (UID: "ab4a4867-caec-44de-bfb4-e9af30e6a7e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:12:22.782962 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.782917    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:12:22.782962 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.782936    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:12:22.784579 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.784558    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab4a4867-caec-44de-bfb4-e9af30e6a7e1" (UID: "ab4a4867-caec-44de-bfb4-e9af30e6a7e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:12:22.784858 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.784839    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-dshm" (OuterVolumeSpecName: "dshm") pod "ab4a4867-caec-44de-bfb4-e9af30e6a7e1" (UID: "ab4a4867-caec-44de-bfb4-e9af30e6a7e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:12:22.784936 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.784913    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-kube-api-access-j8qcm" (OuterVolumeSpecName: "kube-api-access-j8qcm") pod "ab4a4867-caec-44de-bfb4-e9af30e6a7e1" (UID: "ab4a4867-caec-44de-bfb4-e9af30e6a7e1"). InnerVolumeSpecName "kube-api-access-j8qcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:12:22.883899 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.883844    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:12:22.883899 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.883899    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:12:22.884094 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.883914    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:12:22.884094 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:22.883926    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8qcm\" (UniqueName: \"kubernetes.io/projected/ab4a4867-caec-44de-bfb4-e9af30e6a7e1-kube-api-access-j8qcm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:12:23.036937 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:23.036857    2580 generic.go:356] "Generic (PLEG): container finished" podID="ab4a4867-caec-44de-bfb4-e9af30e6a7e1" containerID="fbcf93879a1762c3d1e449019abfc7118b578e4dc5b088099ba4d90d94a49dbc" exitCode=0
Sep 15 12:12:23.036937 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:23.036908    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw" event={"ID":"ab4a4867-caec-44de-bfb4-e9af30e6a7e1","Type":"ContainerDied","Data":"fbcf93879a1762c3d1e449019abfc7118b578e4dc5b088099ba4d90d94a49dbc"}
Sep 15 12:12:23.037203 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:23.036921    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"
Sep 15 12:12:23.037203 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:23.036959    2580 scope.go:117] "RemoveContainer" containerID="fbcf93879a1762c3d1e449019abfc7118b578e4dc5b088099ba4d90d94a49dbc"
Sep 15 12:12:23.037203 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:23.036949    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw" event={"ID":"ab4a4867-caec-44de-bfb4-e9af30e6a7e1","Type":"ContainerDied","Data":"2fa88715b65351b441e1bb82181449a380fa02b2f42aa65dc8186437b5d864b6"}
Sep 15 12:12:23.077513 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:23.077483    2580 scope.go:117] "RemoveContainer" containerID="fbcf93879a1762c3d1e449019abfc7118b578e4dc5b088099ba4d90d94a49dbc"
Sep 15 12:12:23.077894 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:12:23.077848    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbcf93879a1762c3d1e449019abfc7118b578e4dc5b088099ba4d90d94a49dbc\": container with ID starting with fbcf93879a1762c3d1e449019abfc7118b578e4dc5b088099ba4d90d94a49dbc not found: ID does not exist" containerID="fbcf93879a1762c3d1e449019abfc7118b578e4dc5b088099ba4d90d94a49dbc"
Sep 15 12:12:23.078012 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:23.077901    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbcf93879a1762c3d1e449019abfc7118b578e4dc5b088099ba4d90d94a49dbc"} err="failed to get container status \"fbcf93879a1762c3d1e449019abfc7118b578e4dc5b088099ba4d90d94a49dbc\": rpc error: code = NotFound desc = could not find container \"fbcf93879a1762c3d1e449019abfc7118b578e4dc5b088099ba4d90d94a49dbc\": container with ID starting with fbcf93879a1762c3d1e449019abfc7118b578e4dc5b088099ba4d90d94a49dbc not found: ID does not exist"
Sep 15 12:12:23.091462 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:23.091438    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"]
Sep 15 12:12:23.098315 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:23.098295    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32bbbb/rate-test-model-32bbbb-kserve-8b669f79-r2rkw"]
Sep 15 12:12:23.730054 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:23.730022    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="263c9b19-494f-4404-a6aa-a26714006ab9" path="/var/lib/kubelet/pods/263c9b19-494f-4404-a6aa-a26714006ab9/volumes"
Sep 15 12:12:23.730428 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:23.730418    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab4a4867-caec-44de-bfb4-e9af30e6a7e1" path="/var/lib/kubelet/pods/ab4a4867-caec-44de-bfb4-e9af30e6a7e1/volumes"
Sep 15 12:12:28.067633 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:28.067600    2580 generic.go:356] "Generic (PLEG): container finished" podID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerID="7d65bc924864dfa4639221b37f2d1369c85608416ecdf816bfbfe7d7c4955a93" exitCode=0
Sep 15 12:12:28.068154 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:28.067677    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp" event={"ID":"223aa62e-8850-4abe-9961-4ddbb8df9131","Type":"ContainerDied","Data":"7d65bc924864dfa4639221b37f2d1369c85608416ecdf816bfbfe7d7c4955a93"}
Sep 15 12:12:28.068218 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:28.068206    2580 scope.go:117] "RemoveContainer" containerID="7d65bc924864dfa4639221b37f2d1369c85608416ecdf816bfbfe7d7c4955a93"
Sep 15 12:12:28.069288 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:28.069261    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:12:28.069364 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:28.069349    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:12:28.069410 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:28.069397    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:12:29.083639 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:29.083599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp" event={"ID":"223aa62e-8850-4abe-9961-4ddbb8df9131","Type":"ContainerStarted","Data":"ffa5a999865025e08f0fc37159451f36ce7d097856ff16e2afc589de91a9caba"}
Sep 15 12:12:29.470186 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:29.470138    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:29.470404 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:29.470286    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Sep 15 12:12:29.470404 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:29.470331    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp" podUID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Sep 15 12:12:30.470125 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:30.470077    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Sep 15 12:12:30.470626 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:30.470166    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp" podUID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Sep 15 12:12:31.470054 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:31.470014    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Sep 15 12:12:31.470358 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:31.470325    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp" podUID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Sep 15 12:12:32.268194 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:32.268153    2580 scope.go:117] "RemoveContainer" containerID="da2f890e80d371a8df7dc0823e0015cee6a1358c0375ef7a397b818ef50e4176"
Sep 15 12:12:32.278211 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:32.278189    2580 scope.go:117] "RemoveContainer" containerID="28fe1ef5b1dc2fbd8ddcf31905fd9b6c5da4984b00b217f5287de0ed804eef2c"
Sep 15 12:12:32.286023 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:32.286004    2580 scope.go:117] "RemoveContainer" containerID="dcc823f2b44a5e0b2eefda6627c2dbdfa64e5d20ba94ec6bdb2b4bfdfa54b463"
Sep 15 12:12:32.470721 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:32.470656    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Sep 15 12:12:32.471241 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:32.470741    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp" podUID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Sep 15 12:12:33.469598 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:33.469560    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Sep 15 12:12:33.470283 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:33.470249    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp" podUID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Sep 15 12:12:34.470219 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:34.470180    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Sep 15 12:12:34.470597 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:34.470249    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp" podUID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Sep 15 12:12:35.473507 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:35.473472    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:35.473894 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:35.473759    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:35.474522 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:35.474502    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:12:36.863827 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:36.863793    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"]
Sep 15 12:12:36.864214 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:36.864178    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4a4867-caec-44de-bfb4-e9af30e6a7e1" containerName="main"
Sep 15 12:12:36.864214 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:36.864192    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4a4867-caec-44de-bfb4-e9af30e6a7e1" containerName="main"
Sep 15 12:12:36.864214 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:36.864206    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="263c9b19-494f-4404-a6aa-a26714006ab9" containerName="main"
Sep 15 12:12:36.864214 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:36.864211    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="263c9b19-494f-4404-a6aa-a26714006ab9" containerName="main"
Sep 15 12:12:36.864348 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:36.864280    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab4a4867-caec-44de-bfb4-e9af30e6a7e1" containerName="main"
Sep 15 12:12:36.864348 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:36.864290    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="263c9b19-494f-4404-a6aa-a26714006ab9" containerName="main"
Sep 15 12:12:36.866217 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:36.866197    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:36.877382 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:36.877363    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-74aca3-openshift-default-dockercfg-sjl4v\""
Sep 15 12:12:36.905023 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:36.904996    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"]
Sep 15 12:12:37.021074 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.021041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p8mn\" (UniqueName: \"kubernetes.io/projected/1625e331-b139-4d49-91f5-804068870be2-kube-api-access-2p8mn\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.021258 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.021081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1625e331-b139-4d49-91f5-804068870be2-istio-token\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.021258 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.021103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-istio-data\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.021258 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.021183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-credential-socket\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.021258 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.021207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-istio-envoy\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.021258 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.021236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1625e331-b139-4d49-91f5-804068870be2-istio-podinfo\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.021258 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.021259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-workload-socket\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.021495 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.021313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1625e331-b139-4d49-91f5-804068870be2-istiod-ca-cert\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.021495 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.021346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-workload-certs\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.122649 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.122561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1625e331-b139-4d49-91f5-804068870be2-istio-token\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.122649 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.122607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-istio-data\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.122924 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.122650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-credential-socket\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.122924 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.122668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-istio-envoy\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.122924 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.122704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1625e331-b139-4d49-91f5-804068870be2-istio-podinfo\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.122924 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.122742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-workload-socket\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.122924 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.122777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1625e331-b139-4d49-91f5-804068870be2-istiod-ca-cert\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.122924 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.122809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-workload-certs\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.122924 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.122841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p8mn\" (UniqueName: \"kubernetes.io/projected/1625e331-b139-4d49-91f5-804068870be2-kube-api-access-2p8mn\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.123310 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.123162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-istio-data\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.123424 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.123382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-workload-certs\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.123510 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.123484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-workload-socket\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.123573 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.123502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-credential-socket\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.123700 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.123675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1625e331-b139-4d49-91f5-804068870be2-istiod-ca-cert\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.125990 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.125966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1625e331-b139-4d49-91f5-804068870be2-istio-podinfo\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.126083 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.126012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-istio-envoy\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.135315 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.135286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p8mn\" (UniqueName: \"kubernetes.io/projected/1625e331-b139-4d49-91f5-804068870be2-kube-api-access-2p8mn\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.137341 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.137319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1625e331-b139-4d49-91f5-804068870be2-istio-token\") pod \"e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") " pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.176294 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.176268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:37.466516 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.466480    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"]
Sep 15 12:12:37.467511 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:12:37.467483    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1625e331_b139_4d49_91f5_804068870be2.slice/crio-933aecd8f3020a46de82242b89ef1c4c8bc351441bee61658e40c0af6d0a2286 WatchSource:0}: Error finding container 933aecd8f3020a46de82242b89ef1c4c8bc351441bee61658e40c0af6d0a2286: Status 404 returned error can't find the container with id 933aecd8f3020a46de82242b89ef1c4c8bc351441bee61658e40c0af6d0a2286
Sep 15 12:12:37.470104 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.470072    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:12:37.470181 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.470148    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:12:37.470181 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:37.470176    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:12:38.134530 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:38.134490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b" event={"ID":"1625e331-b139-4d49-91f5-804068870be2","Type":"ContainerStarted","Data":"6f482cc84f6737d6c90ddc43bccb88d1a18ea8c9ac3a7ce3e8491f75f9ffc40e"}
Sep 15 12:12:38.134937 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:38.134551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b" event={"ID":"1625e331-b139-4d49-91f5-804068870be2","Type":"ContainerStarted","Data":"933aecd8f3020a46de82242b89ef1c4c8bc351441bee61658e40c0af6d0a2286"}
Sep 15 12:12:38.157952 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:38.157903    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b" podStartSLOduration=2.15785737 podStartE2EDuration="2.15785737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:12:38.156937506 +0000 UTC m=+2046.937508897" watchObservedRunningTime="2026-09-15 12:12:38.15785737 +0000 UTC m=+2046.938428757"
Sep 15 12:12:38.177140 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:38.177113    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:38.182158 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:38.182138    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:39.139859 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:39.139819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:39.140663 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:39.140643    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:12:43.917843 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:43.917809    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"]
Sep 15 12:12:43.920612 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:43.920594    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"
Sep 15 12:12:43.922868 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:43.922848    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-74aca3-dockercfg-tmqk8\""
Sep 15 12:12:43.922868 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:43.922891    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-74aca3\""
Sep 15 12:12:43.931064 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:43.931040    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"]
Sep 15 12:12:43.933615 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:43.933596    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"
Sep 15 12:12:43.934174 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:43.934153    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"]
Sep 15 12:12:43.946278 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:43.946256    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"]
Sep 15 12:12:43.974064 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:43.974037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40741ce9-1889-4d94-b5aa-0f6aeae67156-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh\" (UID: \"40741ce9-1889-4d94-b5aa-0f6aeae67156\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"
Sep 15 12:12:43.974235 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:43.974216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-425hr\" (UniqueName: \"kubernetes.io/projected/40741ce9-1889-4d94-b5aa-0f6aeae67156-kube-api-access-425hr\") pod \"payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh\" (UID: \"40741ce9-1889-4d94-b5aa-0f6aeae67156\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"
Sep 15 12:12:44.081257 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:44.075510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d253f8-9fa5-4c64-8e6f-692bb1da76da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9\" (UID: \"e7d253f8-9fa5-4c64-8e6f-692bb1da76da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"
Sep 15 12:12:44.081257 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:44.075615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40741ce9-1889-4d94-b5aa-0f6aeae67156-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh\" (UID: \"40741ce9-1889-4d94-b5aa-0f6aeae67156\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"
Sep 15 12:12:44.081257 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:44.075675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-425hr\" (UniqueName: \"kubernetes.io/projected/40741ce9-1889-4d94-b5aa-0f6aeae67156-kube-api-access-425hr\") pod \"payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh\" (UID: \"40741ce9-1889-4d94-b5aa-0f6aeae67156\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"
Sep 15 12:12:44.081257 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:44.075741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9cwf\" (UniqueName: \"kubernetes.io/projected/e7d253f8-9fa5-4c64-8e6f-692bb1da76da-kube-api-access-k9cwf\") pod \"payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9\" (UID: \"e7d253f8-9fa5-4c64-8e6f-692bb1da76da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"
Sep 15 12:12:44.081257 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:44.076767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40741ce9-1889-4d94-b5aa-0f6aeae67156-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh\" (UID: \"40741ce9-1889-4d94-b5aa-0f6aeae67156\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"
Sep 15 12:12:44.086534 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:44.086510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-425hr\" (UniqueName: \"kubernetes.io/projected/40741ce9-1889-4d94-b5aa-0f6aeae67156-kube-api-access-425hr\") pod \"payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh\" (UID: \"40741ce9-1889-4d94-b5aa-0f6aeae67156\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"
Sep 15 12:12:44.176502 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:44.176413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d253f8-9fa5-4c64-8e6f-692bb1da76da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9\" (UID: \"e7d253f8-9fa5-4c64-8e6f-692bb1da76da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"
Sep 15 12:12:44.176502 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:44.176498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9cwf\" (UniqueName: \"kubernetes.io/projected/e7d253f8-9fa5-4c64-8e6f-692bb1da76da-kube-api-access-k9cwf\") pod \"payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9\" (UID: \"e7d253f8-9fa5-4c64-8e6f-692bb1da76da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"
Sep 15 12:12:44.177171 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:44.177140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d253f8-9fa5-4c64-8e6f-692bb1da76da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9\" (UID: \"e7d253f8-9fa5-4c64-8e6f-692bb1da76da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"
Sep 15 12:12:44.185166 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:44.185142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9cwf\" (UniqueName: \"kubernetes.io/projected/e7d253f8-9fa5-4c64-8e6f-692bb1da76da-kube-api-access-k9cwf\") pod \"payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9\" (UID: \"e7d253f8-9fa5-4c64-8e6f-692bb1da76da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"
Sep 15 12:12:44.232175 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:44.232144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"
Sep 15 12:12:44.244072 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:44.244044    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"
Sep 15 12:12:44.509185 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:44.509150    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"]
Sep 15 12:12:44.510098 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:12:44.510075    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d253f8_9fa5_4c64_8e6f_692bb1da76da.slice/crio-1e1f96c2c0da95aac700d0828abaa54b8f493b17f312e2cc07ac2a5db6bad95f WatchSource:0}: Error finding container 1e1f96c2c0da95aac700d0828abaa54b8f493b17f312e2cc07ac2a5db6bad95f: Status 404 returned error can't find the container with id 1e1f96c2c0da95aac700d0828abaa54b8f493b17f312e2cc07ac2a5db6bad95f
Sep 15 12:12:44.528164 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:44.528137    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"]
Sep 15 12:12:44.529364 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:12:44.529337    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40741ce9_1889_4d94_b5aa_0f6aeae67156.slice/crio-026a6351c41d43d4e1bc4324a62124b877c9cdb7129c17ec0f08b68f7a58ddc4 WatchSource:0}: Error finding container 026a6351c41d43d4e1bc4324a62124b877c9cdb7129c17ec0f08b68f7a58ddc4: Status 404 returned error can't find the container with id 026a6351c41d43d4e1bc4324a62124b877c9cdb7129c17ec0f08b68f7a58ddc4
Sep 15 12:12:45.173707 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:45.173675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh" event={"ID":"40741ce9-1889-4d94-b5aa-0f6aeae67156","Type":"ContainerStarted","Data":"d1da0853b5df537e7b8d1ed4f1276384f3627d9bc89c582b113b2206aa1a9b95"}
Sep 15 12:12:45.174212 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:45.173736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh" event={"ID":"40741ce9-1889-4d94-b5aa-0f6aeae67156","Type":"ContainerStarted","Data":"026a6351c41d43d4e1bc4324a62124b877c9cdb7129c17ec0f08b68f7a58ddc4"}
Sep 15 12:12:45.174212 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:45.173768    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"
Sep 15 12:12:45.175573 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:45.175542    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9" event={"ID":"e7d253f8-9fa5-4c64-8e6f-692bb1da76da","Type":"ContainerStarted","Data":"14f5eb30d06fe924697a98c14c8b1127612aae6c6276dc141d8bbada1d2f35e1"}
Sep 15 12:12:45.175708 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:45.175589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9" event={"ID":"e7d253f8-9fa5-4c64-8e6f-692bb1da76da","Type":"ContainerStarted","Data":"1e1f96c2c0da95aac700d0828abaa54b8f493b17f312e2cc07ac2a5db6bad95f"}
Sep 15 12:12:45.175708 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:45.175627    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"
Sep 15 12:12:45.192184 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:45.192136    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh" podStartSLOduration=2.192121164 podStartE2EDuration="2.192121164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:12:45.189833662 +0000 UTC m=+2053.970405051" watchObservedRunningTime="2026-09-15 12:12:45.192121164 +0000 UTC m=+2053.972692552"
Sep 15 12:12:45.206821 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:45.206776    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9" podStartSLOduration=2.206761604 podStartE2EDuration="2.206761604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:12:45.20520971 +0000 UTC m=+2053.985781096" watchObservedRunningTime="2026-09-15 12:12:45.206761604 +0000 UTC m=+2053.987332992"
Sep 15 12:12:56.183636 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:56.183601    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"
Sep 15 12:12:56.184087 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:56.183690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"
Sep 15 12:12:58.254539 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:58.254505    2580 generic.go:356] "Generic (PLEG): container finished" podID="1625e331-b139-4d49-91f5-804068870be2" containerID="6f482cc84f6737d6c90ddc43bccb88d1a18ea8c9ac3a7ce3e8491f75f9ffc40e" exitCode=0
Sep 15 12:12:58.255025 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:58.254574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b" event={"ID":"1625e331-b139-4d49-91f5-804068870be2","Type":"ContainerDied","Data":"6f482cc84f6737d6c90ddc43bccb88d1a18ea8c9ac3a7ce3e8491f75f9ffc40e"}
Sep 15 12:12:58.255025 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:58.254984    2580 scope.go:117] "RemoveContainer" containerID="6f482cc84f6737d6c90ddc43bccb88d1a18ea8c9ac3a7ce3e8491f75f9ffc40e"
Sep 15 12:12:58.256097 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:58.256054    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:12:58.256185 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:58.256140    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:12:58.256185 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:58.256174    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:12:59.265867 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:12:59.265821    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b" event={"ID":"1625e331-b139-4d49-91f5-804068870be2","Type":"ContainerStarted","Data":"f48ac6f28c39cc5fd3c69b76880091a71500f7d307f672ab6570bf30f43863d5"}
Sep 15 12:13:00.177233 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:00.177198    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:13:00.177534 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:00.177512    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Sep 15 12:13:00.177586 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:00.177564    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b" podUID="1625e331-b139-4d49-91f5-804068870be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Sep 15 12:13:01.177619 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:01.177577    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Sep 15 12:13:01.178062 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:01.177643    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b" podUID="1625e331-b139-4d49-91f5-804068870be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Sep 15 12:13:02.177413 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:02.177363    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Sep 15 12:13:02.177649 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:02.177420    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b" podUID="1625e331-b139-4d49-91f5-804068870be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Sep 15 12:13:03.176827 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:03.176765    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Sep 15 12:13:03.177030 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:03.176861    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b" podUID="1625e331-b139-4d49-91f5-804068870be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Sep 15 12:13:04.177451 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:04.177412    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Sep 15 12:13:04.177886 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:04.177468    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b" podUID="1625e331-b139-4d49-91f5-804068870be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Sep 15 12:13:05.181441 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:05.181408    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:13:05.181848 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:05.181677    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:13:05.182305 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:05.182289    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:13:15.247838 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:15.247805    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-j556w_b9653379-1bc2-4fb8-9c12-9a92ca8eb998/payload-processing/0.log"
Sep 15 12:13:15.363784 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:15.363743    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh_40741ce9-1889-4d94-b5aa-0f6aeae67156/payload-processing/0.log"
Sep 15 12:13:26.990198 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:26.990163    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"]
Sep 15 12:13:26.993908 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:26.993887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:26.999283 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:26.999255    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7139fb\"/\"ipp-route-7139fb-kserve-self-signed-certs\""
Sep 15 12:13:26.999283 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:26.999279    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7139fb\"/\"default-dockercfg-dg2s4\""
Sep 15 12:13:26.999787 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:26.999767    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7139fb\"/\"openshift-service-ca.crt\""
Sep 15 12:13:27.000061 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.000045    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7139fb\"/\"kube-root-ca.crt\""
Sep 15 12:13:27.007775 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.007748    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"]
Sep 15 12:13:27.170347 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.170312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d5a887-6116-47d5-8b44-fdda083f40bd-tls-certs\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.170506 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.170365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-tmp-dir\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.170506 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.170449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-dshm\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.170506 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.170487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-model-cache\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.170669 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.170510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgw86\" (UniqueName: \"kubernetes.io/projected/66d5a887-6116-47d5-8b44-fdda083f40bd-kube-api-access-xgw86\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.170669 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.170573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-home\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.271576 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.271490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-tmp-dir\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.271576 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.271555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-dshm\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.271807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.271594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-model-cache\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.271807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.271624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgw86\" (UniqueName: \"kubernetes.io/projected/66d5a887-6116-47d5-8b44-fdda083f40bd-kube-api-access-xgw86\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.271807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.271661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-home\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.271807 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.271709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d5a887-6116-47d5-8b44-fdda083f40bd-tls-certs\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.272097 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.272009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-tmp-dir\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.272319 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.272294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-model-cache\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.272730 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.272707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-home\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.274708 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.274685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-dshm\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.275141 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.275115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d5a887-6116-47d5-8b44-fdda083f40bd-tls-certs\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.284748 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.284728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgw86\" (UniqueName: \"kubernetes.io/projected/66d5a887-6116-47d5-8b44-fdda083f40bd-kube-api-access-xgw86\") pod \"ipp-route-7139fb-kserve-58c65cf84c-krgkn\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") " pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.303620 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.303600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:27.582134 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:27.582110    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"]
Sep 15 12:13:27.583715 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:13:27.583685    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66d5a887_6116_47d5_8b44_fdda083f40bd.slice/crio-4723214691a89cf10489d8ba59a4be98c325314f40469b5c0cb19a3d0a89bdbb WatchSource:0}: Error finding container 4723214691a89cf10489d8ba59a4be98c325314f40469b5c0cb19a3d0a89bdbb: Status 404 returned error can't find the container with id 4723214691a89cf10489d8ba59a4be98c325314f40469b5c0cb19a3d0a89bdbb
Sep 15 12:13:28.424852 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:28.424817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn" event={"ID":"66d5a887-6116-47d5-8b44-fdda083f40bd","Type":"ContainerStarted","Data":"f168de2274d63d179eed5fa7ad7023ab4cac028f3c841b2c69bf6496a8d5df26"}
Sep 15 12:13:28.425253 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:28.424858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn" event={"ID":"66d5a887-6116-47d5-8b44-fdda083f40bd","Type":"ContainerStarted","Data":"4723214691a89cf10489d8ba59a4be98c325314f40469b5c0cb19a3d0a89bdbb"}
Sep 15 12:13:28.450156 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:28.450111    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn" podStartSLOduration=2.450096592 podStartE2EDuration="2.450096592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:13:28.44762545 +0000 UTC m=+2097.228196842" watchObservedRunningTime="2026-09-15 12:13:28.450096592 +0000 UTC m=+2097.230667980"
Sep 15 12:13:32.318824 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:32.318792    2580 scope.go:117] "RemoveContainer" containerID="080ccf166b3d5956474a5165c88a4c254a20ceb887ac1484bf45de6b2d6479f5"
Sep 15 12:13:37.304504 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:37.304469    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:37.304961 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:37.304524    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:37.316987 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:37.316961    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:37.483975 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:37.483942    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:13:53.648059 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:53.648024    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-j556w_b9653379-1bc2-4fb8-9c12-9a92ca8eb998/payload-processing/0.log"
Sep 15 12:13:53.769537 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:53.769508    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh_40741ce9-1889-4d94-b5aa-0f6aeae67156/payload-processing/0.log"
Sep 15 12:13:53.894929 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:13:53.894898    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh_40741ce9-1889-4d94-b5aa-0f6aeae67156/payload-processing/0.log"
Sep 15 12:14:28.996591 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:28.996517    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9"]
Sep 15 12:14:28.999951 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:28.999929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9"
Sep 15 12:14:29.002567 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:29.002540    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:14:29.002667 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:29.002636    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:14:29.009706 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:29.009678    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9"]
Sep 15 12:14:29.135261 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:29.135221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9\" (UID: \"3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9"
Sep 15 12:14:29.236162 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:29.236127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9\" (UID: \"3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9"
Sep 15 12:14:29.236703 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:29.236683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9\" (UID: \"3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9"
Sep 15 12:14:29.311362 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:29.311270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9"
Sep 15 12:14:29.586192 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:29.586094    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9"]
Sep 15 12:14:29.588547 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:14:29.588518    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c16ecf3_aff3_4de1_b77c_b4c8c0f8eb01.slice/crio-022a345b815b20c79ee39c7b216c07b7c2c87ef06e455e8519aacd912a78b0ec WatchSource:0}: Error finding container 022a345b815b20c79ee39c7b216c07b7c2c87ef06e455e8519aacd912a78b0ec: Status 404 returned error can't find the container with id 022a345b815b20c79ee39c7b216c07b7c2c87ef06e455e8519aacd912a78b0ec
Sep 15 12:14:29.756804 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:29.756766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9" event={"ID":"3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01","Type":"ContainerStarted","Data":"022a345b815b20c79ee39c7b216c07b7c2c87ef06e455e8519aacd912a78b0ec"}
Sep 15 12:14:30.763509 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:30.763469    2580 generic.go:356] "Generic (PLEG): container finished" podID="3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01" containerID="35dc07f35411367769fc5aa6bad40e46f6c09aaf3a87242f4944f5ef9df1a306" exitCode=0
Sep 15 12:14:30.763925 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:30.763549    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9" event={"ID":"3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01","Type":"ContainerDied","Data":"35dc07f35411367769fc5aa6bad40e46f6c09aaf3a87242f4944f5ef9df1a306"}
Sep 15 12:14:31.898000 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:31.897977    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9"
Sep 15 12:14:31.959519 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:31.959493    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01-maas-api-service-ca\") pod \"3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01\" (UID: \"3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01\") "
Sep 15 12:14:31.959896 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:31.959853    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01" (UID: "3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:14:32.061307 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:32.061211    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:14:32.339402 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:32.339313    2580 scope.go:117] "RemoveContainer" containerID="f774ad5a5a0c035b55b504b13ce9ac254ea2fbdf17673d65e9bd05a317be65a2"
Sep 15 12:14:32.774687 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:32.774650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9" event={"ID":"3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01","Type":"ContainerDied","Data":"022a345b815b20c79ee39c7b216c07b7c2c87ef06e455e8519aacd912a78b0ec"}
Sep 15 12:14:32.774687 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:32.774677    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9"
Sep 15 12:14:32.774687 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:32.774694    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="022a345b815b20c79ee39c7b216c07b7c2c87ef06e455e8519aacd912a78b0ec"
Sep 15 12:14:40.794154 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:40.794115    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9"]
Sep 15 12:14:40.797112 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:40.797083    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fd646-ckrt9"]
Sep 15 12:14:41.732032 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:41.731991    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01" path="/var/lib/kubelet/pods/3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01/volumes"
Sep 15 12:14:51.694577 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:51.693023    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v"]
Sep 15 12:14:51.694577 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:51.693821    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01" containerName="revoke-keys"
Sep 15 12:14:51.694577 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:51.693843    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01" containerName="revoke-keys"
Sep 15 12:14:51.694577 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:51.694059    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c16ecf3-aff3-4de1-b77c-b4c8c0f8eb01" containerName="revoke-keys"
Sep 15 12:14:51.697925 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:51.697906    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v"
Sep 15 12:14:51.703057 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:51.703006    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:14:51.703272 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:51.703254    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:14:51.707435 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:51.707412    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v"]
Sep 15 12:14:51.845654 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:51.845615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed348b0-2e69-4aaa-9553-2769205490d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v\" (UID: \"0ed348b0-2e69-4aaa-9553-2769205490d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v"
Sep 15 12:14:51.946383 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:51.946296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed348b0-2e69-4aaa-9553-2769205490d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v\" (UID: \"0ed348b0-2e69-4aaa-9553-2769205490d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v"
Sep 15 12:14:51.947001 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:51.946979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed348b0-2e69-4aaa-9553-2769205490d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v\" (UID: \"0ed348b0-2e69-4aaa-9553-2769205490d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v"
Sep 15 12:14:52.009323 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:52.009291    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v"
Sep 15 12:14:52.282392 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:52.282360    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v"]
Sep 15 12:14:52.284021 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:14:52.283987    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed348b0_2e69_4aaa_9553_2769205490d4.slice/crio-f0a5c33f459a7b15b3d24f6d616c83827d4e05cda6831717ea518e1096e3ff7f WatchSource:0}: Error finding container f0a5c33f459a7b15b3d24f6d616c83827d4e05cda6831717ea518e1096e3ff7f: Status 404 returned error can't find the container with id f0a5c33f459a7b15b3d24f6d616c83827d4e05cda6831717ea518e1096e3ff7f
Sep 15 12:14:52.881540 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:52.881450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v" event={"ID":"0ed348b0-2e69-4aaa-9553-2769205490d4","Type":"ContainerStarted","Data":"08219b90d724fd4d35bfd3cc8d4a66a2209234afc0240bd30a6fa541e2ddf54a"}
Sep 15 12:14:52.881540 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:52.881514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v" event={"ID":"0ed348b0-2e69-4aaa-9553-2769205490d4","Type":"ContainerStarted","Data":"f0a5c33f459a7b15b3d24f6d616c83827d4e05cda6831717ea518e1096e3ff7f"}
Sep 15 12:14:52.904811 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:52.904744    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v" podStartSLOduration=1.904726449 podStartE2EDuration="1.904726449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:14:52.90195198 +0000 UTC m=+2181.682523368" watchObservedRunningTime="2026-09-15 12:14:52.904726449 +0000 UTC m=+2181.685297837"
Sep 15 12:14:53.887987 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:53.887951    2580 generic.go:356] "Generic (PLEG): container finished" podID="0ed348b0-2e69-4aaa-9553-2769205490d4" containerID="08219b90d724fd4d35bfd3cc8d4a66a2209234afc0240bd30a6fa541e2ddf54a" exitCode=0
Sep 15 12:14:53.888368 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:53.888038    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v" event={"ID":"0ed348b0-2e69-4aaa-9553-2769205490d4","Type":"ContainerDied","Data":"08219b90d724fd4d35bfd3cc8d4a66a2209234afc0240bd30a6fa541e2ddf54a"}
Sep 15 12:14:55.023035 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:55.023009    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v"
Sep 15 12:14:55.073981 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:55.073951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed348b0-2e69-4aaa-9553-2769205490d4-maas-api-service-ca\") pod \"0ed348b0-2e69-4aaa-9553-2769205490d4\" (UID: \"0ed348b0-2e69-4aaa-9553-2769205490d4\") "
Sep 15 12:14:55.074386 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:55.074360    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed348b0-2e69-4aaa-9553-2769205490d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ed348b0-2e69-4aaa-9553-2769205490d4" (UID: "0ed348b0-2e69-4aaa-9553-2769205490d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:14:55.174734 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:55.174690    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed348b0-2e69-4aaa-9553-2769205490d4-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:14:55.902731 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:55.902694    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v"
Sep 15 12:14:55.902908 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:55.902731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v" event={"ID":"0ed348b0-2e69-4aaa-9553-2769205490d4","Type":"ContainerDied","Data":"f0a5c33f459a7b15b3d24f6d616c83827d4e05cda6831717ea518e1096e3ff7f"}
Sep 15 12:14:55.902908 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:14:55.902803    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0a5c33f459a7b15b3d24f6d616c83827d4e05cda6831717ea518e1096e3ff7f"
Sep 15 12:15:00.148305 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.148267    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32bbbb-29824575-sxlkr"]
Sep 15 12:15:00.148711 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.148668    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ed348b0-2e69-4aaa-9553-2769205490d4" containerName="revoke-keys"
Sep 15 12:15:00.148711 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.148681    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ed348b0-2e69-4aaa-9553-2769205490d4" containerName="revoke-keys"
Sep 15 12:15:00.148787 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.148754    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ed348b0-2e69-4aaa-9553-2769205490d4" containerName="revoke-keys"
Sep 15 12:15:00.152014 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.151988    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32bbbb-29824575-sxlkr"
Sep 15 12:15:00.154043 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.154018    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b3936-29824575-zt4mq"]
Sep 15 12:15:00.157256 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.157237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b3936-29824575-zt4mq"
Sep 15 12:15:00.159362 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.159346    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824575-djw7l"]
Sep 15 12:15:00.162934 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.162908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824575-djw7l"
Sep 15 12:15:00.165038 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.165019    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:15:00.171528 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.171504    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b3936-29824575-zt4mq"
Sep 15 12:15:00.174305 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.174289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32bbbb-29824575-sxlkr"
Sep 15 12:15:00.174892 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.174745    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32bbbb-29824575-sxlkr"]
Sep 15 12:15:00.180378 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.180359    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824575-djw7l"
Sep 15 12:15:00.180532 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.180515    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b3936-29824575-zt4mq"]
Sep 15 12:15:00.188910 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.188886    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824575-djw7l"]
Sep 15 12:15:00.477897 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.477839    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b3936-29824575-zt4mq"]
Sep 15 12:15:00.479642 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:15:00.479611    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4932a671_5d5e_4d37_91ae_9583064091e9.slice/crio-df8a502701bd3c0a1209a7f418794f34080a3b387b952366b0b942f8afe69425 WatchSource:0}: Error finding container df8a502701bd3c0a1209a7f418794f34080a3b387b952366b0b942f8afe69425: Status 404 returned error can't find the container with id df8a502701bd3c0a1209a7f418794f34080a3b387b952366b0b942f8afe69425
Sep 15 12:15:00.509134 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.509110    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32bbbb-29824575-sxlkr"]
Sep 15 12:15:00.512260 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:15:00.512210    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b23e5f5_9958_40e5_82e4_707bc968dbd2.slice/crio-35e1df8e391beaf39651adf17e987649220f570b8d2c8a884f3f14e8fe0d7d8a WatchSource:0}: Error finding container 35e1df8e391beaf39651adf17e987649220f570b8d2c8a884f3f14e8fe0d7d8a: Status 404 returned error can't find the container with id 35e1df8e391beaf39651adf17e987649220f570b8d2c8a884f3f14e8fe0d7d8a
Sep 15 12:15:00.515274 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.515255    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824575-djw7l"]
Sep 15 12:15:00.516609 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:15:00.516583    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372c3ac5_4781_48bd_9faa_5b43e5b2f02d.slice/crio-369316d8c17903e4a37f598bb1c65715407663303ccf6aeb3d78a4e1b0443fb2 WatchSource:0}: Error finding container 369316d8c17903e4a37f598bb1c65715407663303ccf6aeb3d78a4e1b0443fb2: Status 404 returned error can't find the container with id 369316d8c17903e4a37f598bb1c65715407663303ccf6aeb3d78a4e1b0443fb2
Sep 15 12:15:00.934553 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.934509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824575-djw7l" event={"ID":"372c3ac5-4781-48bd-9faa-5b43e5b2f02d","Type":"ContainerStarted","Data":"369316d8c17903e4a37f598bb1c65715407663303ccf6aeb3d78a4e1b0443fb2"}
Sep 15 12:15:00.936070 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.936033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32bbbb-29824575-sxlkr" event={"ID":"1b23e5f5-9958-40e5-82e4-707bc968dbd2","Type":"ContainerStarted","Data":"f3a2bf16f7a4cb6c3c3d4efcc2b57e90ca17080914c9e6274baba3dea24000bd"}
Sep 15 12:15:00.936197 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.936091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32bbbb-29824575-sxlkr" event={"ID":"1b23e5f5-9958-40e5-82e4-707bc968dbd2","Type":"ContainerStarted","Data":"35e1df8e391beaf39651adf17e987649220f570b8d2c8a884f3f14e8fe0d7d8a"}
Sep 15 12:15:00.937478 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.937454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b3936-29824575-zt4mq" event={"ID":"4932a671-5d5e-4d37-91ae-9583064091e9","Type":"ContainerStarted","Data":"90e16a258565588c07e1eab9796e66ee83ea5d24b6c6855c1daff3a99b3d7753"}
Sep 15 12:15:00.937577 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.937494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b3936-29824575-zt4mq" event={"ID":"4932a671-5d5e-4d37-91ae-9583064091e9","Type":"ContainerStarted","Data":"df8a502701bd3c0a1209a7f418794f34080a3b387b952366b0b942f8afe69425"}
Sep 15 12:15:00.962415 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.962367    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32bbbb-29824575-sxlkr" podStartSLOduration=0.962351602 podStartE2EDuration="962.351602ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:00.960596815 +0000 UTC m=+2189.741168199" watchObservedRunningTime="2026-09-15 12:15:00.962351602 +0000 UTC m=+2189.742922989"
Sep 15 12:15:00.986280 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:00.986220    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b3936-29824575-zt4mq" podStartSLOduration=0.986204604 podStartE2EDuration="986.204604ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:00.984955794 +0000 UTC m=+2189.765527182" watchObservedRunningTime="2026-09-15 12:15:00.986204604 +0000 UTC m=+2189.766776216"
Sep 15 12:15:01.944816 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:01.944780    2580 generic.go:356] "Generic (PLEG): container finished" podID="1b23e5f5-9958-40e5-82e4-707bc968dbd2" containerID="f3a2bf16f7a4cb6c3c3d4efcc2b57e90ca17080914c9e6274baba3dea24000bd" exitCode=0
Sep 15 12:15:01.945253 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:01.944820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32bbbb-29824575-sxlkr" event={"ID":"1b23e5f5-9958-40e5-82e4-707bc968dbd2","Type":"ContainerDied","Data":"f3a2bf16f7a4cb6c3c3d4efcc2b57e90ca17080914c9e6274baba3dea24000bd"}
Sep 15 12:15:01.946476 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:01.946454    2580 generic.go:356] "Generic (PLEG): container finished" podID="4932a671-5d5e-4d37-91ae-9583064091e9" containerID="90e16a258565588c07e1eab9796e66ee83ea5d24b6c6855c1daff3a99b3d7753" exitCode=0
Sep 15 12:15:01.946570 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:01.946533    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b3936-29824575-zt4mq" event={"ID":"4932a671-5d5e-4d37-91ae-9583064091e9","Type":"ContainerDied","Data":"90e16a258565588c07e1eab9796e66ee83ea5d24b6c6855c1daff3a99b3d7753"}
Sep 15 12:15:01.948087 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:01.948068    2580 generic.go:356] "Generic (PLEG): container finished" podID="372c3ac5-4781-48bd-9faa-5b43e5b2f02d" containerID="77b2748e02ea1517dcfb02bea1cc4486d2fa31880a8548d1f41438e8d02093ff" exitCode=0
Sep 15 12:15:01.948195 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:01.948132    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824575-djw7l" event={"ID":"372c3ac5-4781-48bd-9faa-5b43e5b2f02d","Type":"ContainerDied","Data":"77b2748e02ea1517dcfb02bea1cc4486d2fa31880a8548d1f41438e8d02093ff"}
Sep 15 12:15:03.087849 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:03.087819    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b3936-29824575-zt4mq"
Sep 15 12:15:03.134229 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:03.134201    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824575-djw7l"
Sep 15 12:15:03.165201 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:03.165175    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32bbbb-29824575-sxlkr"
Sep 15 12:15:03.961142 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:03.961112    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32bbbb-29824575-sxlkr"
Sep 15 12:15:03.961343 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:03.961113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32bbbb-29824575-sxlkr" event={"ID":"1b23e5f5-9958-40e5-82e4-707bc968dbd2","Type":"ContainerDied","Data":"35e1df8e391beaf39651adf17e987649220f570b8d2c8a884f3f14e8fe0d7d8a"}
Sep 15 12:15:03.961343 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:03.961247    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35e1df8e391beaf39651adf17e987649220f570b8d2c8a884f3f14e8fe0d7d8a"
Sep 15 12:15:03.964012 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:03.963988    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b3936-29824575-zt4mq"
Sep 15 12:15:03.964154 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:03.964029    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b3936-29824575-zt4mq" event={"ID":"4932a671-5d5e-4d37-91ae-9583064091e9","Type":"ContainerDied","Data":"df8a502701bd3c0a1209a7f418794f34080a3b387b952366b0b942f8afe69425"}
Sep 15 12:15:03.964154 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:03.964080    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df8a502701bd3c0a1209a7f418794f34080a3b387b952366b0b942f8afe69425"
Sep 15 12:15:03.965641 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:03.965588    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824575-djw7l"
Sep 15 12:15:03.965641 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:03.965599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824575-djw7l" event={"ID":"372c3ac5-4781-48bd-9faa-5b43e5b2f02d","Type":"ContainerDied","Data":"369316d8c17903e4a37f598bb1c65715407663303ccf6aeb3d78a4e1b0443fb2"}
Sep 15 12:15:03.965641 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:03.965642    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="369316d8c17903e4a37f598bb1c65715407663303ccf6aeb3d78a4e1b0443fb2"
Sep 15 12:15:08.058106 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:08.058071    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v"]
Sep 15 12:15:08.062067 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:08.062035    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7139fb-lcd9v"]
Sep 15 12:15:09.730701 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:09.730665    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ed348b0-2e69-4aaa-9553-2769205490d4" path="/var/lib/kubelet/pods/0ed348b0-2e69-4aaa-9553-2769205490d4/volumes"
Sep 15 12:15:13.428752 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:13.428717    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"]
Sep 15 12:15:28.447939 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.447867    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn" podUID="66d5a887-6116-47d5-8b44-fdda083f40bd" containerName="main" containerID="cri-o://f168de2274d63d179eed5fa7ad7023ab4cac028f3c841b2c69bf6496a8d5df26" gracePeriod=45
Sep 15 12:15:28.696263 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.696236    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:15:28.796013 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.795918    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-model-cache\") pod \"66d5a887-6116-47d5-8b44-fdda083f40bd\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") "
Sep 15 12:15:28.796013 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.795976    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-tmp-dir\") pod \"66d5a887-6116-47d5-8b44-fdda083f40bd\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") "
Sep 15 12:15:28.796013 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.796003    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgw86\" (UniqueName: \"kubernetes.io/projected/66d5a887-6116-47d5-8b44-fdda083f40bd-kube-api-access-xgw86\") pod \"66d5a887-6116-47d5-8b44-fdda083f40bd\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") "
Sep 15 12:15:28.796300 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.796064    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-dshm\") pod \"66d5a887-6116-47d5-8b44-fdda083f40bd\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") "
Sep 15 12:15:28.796300 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.796114    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d5a887-6116-47d5-8b44-fdda083f40bd-tls-certs\") pod \"66d5a887-6116-47d5-8b44-fdda083f40bd\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") "
Sep 15 12:15:28.796300 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.796131    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-home\") pod \"66d5a887-6116-47d5-8b44-fdda083f40bd\" (UID: \"66d5a887-6116-47d5-8b44-fdda083f40bd\") "
Sep 15 12:15:28.796300 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.796248    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "66d5a887-6116-47d5-8b44-fdda083f40bd" (UID: "66d5a887-6116-47d5-8b44-fdda083f40bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:15:28.796300 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.796271    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66d5a887-6116-47d5-8b44-fdda083f40bd" (UID: "66d5a887-6116-47d5-8b44-fdda083f40bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:15:28.796554 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.796446    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-home" (OuterVolumeSpecName: "home") pod "66d5a887-6116-47d5-8b44-fdda083f40bd" (UID: "66d5a887-6116-47d5-8b44-fdda083f40bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:15:28.796554 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.796484    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:15:28.796554 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.796496    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:15:28.798320 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.798298    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-dshm" (OuterVolumeSpecName: "dshm") pod "66d5a887-6116-47d5-8b44-fdda083f40bd" (UID: "66d5a887-6116-47d5-8b44-fdda083f40bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:15:28.798421 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.798378    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d5a887-6116-47d5-8b44-fdda083f40bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66d5a887-6116-47d5-8b44-fdda083f40bd" (UID: "66d5a887-6116-47d5-8b44-fdda083f40bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:15:28.798421 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.798377    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d5a887-6116-47d5-8b44-fdda083f40bd-kube-api-access-xgw86" (OuterVolumeSpecName: "kube-api-access-xgw86") pod "66d5a887-6116-47d5-8b44-fdda083f40bd" (UID: "66d5a887-6116-47d5-8b44-fdda083f40bd"). InnerVolumeSpecName "kube-api-access-xgw86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:15:28.897941 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.897909    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:15:28.898115 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.897954    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d5a887-6116-47d5-8b44-fdda083f40bd-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:15:28.898115 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.897964    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d5a887-6116-47d5-8b44-fdda083f40bd-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:15:28.898115 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:28.897972    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgw86\" (UniqueName: \"kubernetes.io/projected/66d5a887-6116-47d5-8b44-fdda083f40bd-kube-api-access-xgw86\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:15:29.107886 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:29.107779    2580 generic.go:356] "Generic (PLEG): container finished" podID="66d5a887-6116-47d5-8b44-fdda083f40bd" containerID="f168de2274d63d179eed5fa7ad7023ab4cac028f3c841b2c69bf6496a8d5df26" exitCode=0
Sep 15 12:15:29.107886 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:29.107832    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"
Sep 15 12:15:29.108152 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:29.107867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn" event={"ID":"66d5a887-6116-47d5-8b44-fdda083f40bd","Type":"ContainerDied","Data":"f168de2274d63d179eed5fa7ad7023ab4cac028f3c841b2c69bf6496a8d5df26"}
Sep 15 12:15:29.108152 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:29.107934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn" event={"ID":"66d5a887-6116-47d5-8b44-fdda083f40bd","Type":"ContainerDied","Data":"4723214691a89cf10489d8ba59a4be98c325314f40469b5c0cb19a3d0a89bdbb"}
Sep 15 12:15:29.108152 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:29.107953    2580 scope.go:117] "RemoveContainer" containerID="f168de2274d63d179eed5fa7ad7023ab4cac028f3c841b2c69bf6496a8d5df26"
Sep 15 12:15:29.117770 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:29.117750    2580 scope.go:117] "RemoveContainer" containerID="f168de2274d63d179eed5fa7ad7023ab4cac028f3c841b2c69bf6496a8d5df26"
Sep 15 12:15:29.118070 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:15:29.118051    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f168de2274d63d179eed5fa7ad7023ab4cac028f3c841b2c69bf6496a8d5df26\": container with ID starting with f168de2274d63d179eed5fa7ad7023ab4cac028f3c841b2c69bf6496a8d5df26 not found: ID does not exist" containerID="f168de2274d63d179eed5fa7ad7023ab4cac028f3c841b2c69bf6496a8d5df26"
Sep 15 12:15:29.118133 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:29.118078    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f168de2274d63d179eed5fa7ad7023ab4cac028f3c841b2c69bf6496a8d5df26"} err="failed to get container status \"f168de2274d63d179eed5fa7ad7023ab4cac028f3c841b2c69bf6496a8d5df26\": rpc error: code = NotFound desc = could not find container \"f168de2274d63d179eed5fa7ad7023ab4cac028f3c841b2c69bf6496a8d5df26\": container with ID starting with f168de2274d63d179eed5fa7ad7023ab4cac028f3c841b2c69bf6496a8d5df26 not found: ID does not exist"
Sep 15 12:15:29.132101 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:29.132078    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"]
Sep 15 12:15:29.136319 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:29.136300    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-7139fb/ipp-route-7139fb-kserve-58c65cf84c-krgkn"]
Sep 15 12:15:29.730464 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:29.730430    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66d5a887-6116-47d5-8b44-fdda083f40bd" path="/var/lib/kubelet/pods/66d5a887-6116-47d5-8b44-fdda083f40bd/volumes"
Sep 15 12:15:59.785825 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:59.785744    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"]
Sep 15 12:15:59.786346 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:15:59.785994    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp" podUID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerName="istio-proxy" containerID="cri-o://ffa5a999865025e08f0fc37159451f36ce7d097856ff16e2afc589de91a9caba" gracePeriod=30
Sep 15 12:16:00.238359 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.238324    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz"]
Sep 15 12:16:00.238737 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.238716    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="372c3ac5-4781-48bd-9faa-5b43e5b2f02d" containerName="cleanup"
Sep 15 12:16:00.238794 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.238738    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="372c3ac5-4781-48bd-9faa-5b43e5b2f02d" containerName="cleanup"
Sep 15 12:16:00.238794 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.238755    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4932a671-5d5e-4d37-91ae-9583064091e9" containerName="cleanup"
Sep 15 12:16:00.238794 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.238760    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4932a671-5d5e-4d37-91ae-9583064091e9" containerName="cleanup"
Sep 15 12:16:00.238794 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.238768    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b23e5f5-9958-40e5-82e4-707bc968dbd2" containerName="cleanup"
Sep 15 12:16:00.238794 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.238773    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b23e5f5-9958-40e5-82e4-707bc968dbd2" containerName="cleanup"
Sep 15 12:16:00.238794 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.238783    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d5a887-6116-47d5-8b44-fdda083f40bd" containerName="main"
Sep 15 12:16:00.238794 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.238789    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d5a887-6116-47d5-8b44-fdda083f40bd" containerName="main"
Sep 15 12:16:00.239071 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.238848    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b23e5f5-9958-40e5-82e4-707bc968dbd2" containerName="cleanup"
Sep 15 12:16:00.239071 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.238857    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="372c3ac5-4781-48bd-9faa-5b43e5b2f02d" containerName="cleanup"
Sep 15 12:16:00.239071 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.238865    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4932a671-5d5e-4d37-91ae-9583064091e9" containerName="cleanup"
Sep 15 12:16:00.239071 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.238885    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d5a887-6116-47d5-8b44-fdda083f40bd" containerName="main"
Sep 15 12:16:00.241929 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.241914    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz"
Sep 15 12:16:00.247906 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.247883    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:16:00.248199 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.248174    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:16:00.256994 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.256970    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz"]
Sep 15 12:16:00.369848 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.369818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a55ebfca-f406-4115-adfe-a967162d4abf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz\" (UID: \"a55ebfca-f406-4115-adfe-a967162d4abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz"
Sep 15 12:16:00.470938 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.470905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a55ebfca-f406-4115-adfe-a967162d4abf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz\" (UID: \"a55ebfca-f406-4115-adfe-a967162d4abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz"
Sep 15 12:16:00.471515 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.471496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a55ebfca-f406-4115-adfe-a967162d4abf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz\" (UID: \"a55ebfca-f406-4115-adfe-a967162d4abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz"
Sep 15 12:16:00.552297 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.552222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz"
Sep 15 12:16:00.840895 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:00.840854    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz"]
Sep 15 12:16:00.843070 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:16:00.843041    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55ebfca_f406_4115_adfe_a967162d4abf.slice/crio-e5ea6c5c669417b3183923c9c6121fb2959751f3b4eb31e685a556ea3f01009c WatchSource:0}: Error finding container e5ea6c5c669417b3183923c9c6121fb2959751f3b4eb31e685a556ea3f01009c: Status 404 returned error can't find the container with id e5ea6c5c669417b3183923c9c6121fb2959751f3b4eb31e685a556ea3f01009c
Sep 15 12:16:01.297812 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:01.297777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz" event={"ID":"a55ebfca-f406-4115-adfe-a967162d4abf","Type":"ContainerStarted","Data":"a11c795e6c16eb75b15e6678a2c08c94dd24755f1050709785fcfd595ccc0762"}
Sep 15 12:16:01.297989 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:01.297862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz" event={"ID":"a55ebfca-f406-4115-adfe-a967162d4abf","Type":"ContainerStarted","Data":"e5ea6c5c669417b3183923c9c6121fb2959751f3b4eb31e685a556ea3f01009c"}
Sep 15 12:16:01.328560 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:01.328505    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz" podStartSLOduration=1.328489425 podStartE2EDuration="1.328489425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:16:01.328194363 +0000 UTC m=+2250.108765752" watchObservedRunningTime="2026-09-15 12:16:01.328489425 +0000 UTC m=+2250.109060813"
Sep 15 12:16:02.304632 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:02.304594    2580 generic.go:356] "Generic (PLEG): container finished" podID="a55ebfca-f406-4115-adfe-a967162d4abf" containerID="a11c795e6c16eb75b15e6678a2c08c94dd24755f1050709785fcfd595ccc0762" exitCode=0
Sep 15 12:16:02.305117 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:02.304691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz" event={"ID":"a55ebfca-f406-4115-adfe-a967162d4abf","Type":"ContainerDied","Data":"a11c795e6c16eb75b15e6678a2c08c94dd24755f1050709785fcfd595ccc0762"}
Sep 15 12:16:03.438313 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:03.438290    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz"
Sep 15 12:16:03.502513 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:03.502471    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a55ebfca-f406-4115-adfe-a967162d4abf-maas-api-service-ca\") pod \"a55ebfca-f406-4115-adfe-a967162d4abf\" (UID: \"a55ebfca-f406-4115-adfe-a967162d4abf\") "
Sep 15 12:16:03.502857 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:03.502838    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55ebfca-f406-4115-adfe-a967162d4abf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a55ebfca-f406-4115-adfe-a967162d4abf" (UID: "a55ebfca-f406-4115-adfe-a967162d4abf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:03.603294 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:03.603214    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a55ebfca-f406-4115-adfe-a967162d4abf-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:04.318563 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:04.318523    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz" event={"ID":"a55ebfca-f406-4115-adfe-a967162d4abf","Type":"ContainerDied","Data":"e5ea6c5c669417b3183923c9c6121fb2959751f3b4eb31e685a556ea3f01009c"}
Sep 15 12:16:04.318779 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:04.318584    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5ea6c5c669417b3183923c9c6121fb2959751f3b4eb31e685a556ea3f01009c"
Sep 15 12:16:04.318779 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:04.318538    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz"
Sep 15 12:16:05.037180 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.037153    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:16:05.117349 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.117273    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-envoy\") pod \"223aa62e-8850-4abe-9961-4ddbb8df9131\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") "
Sep 15 12:16:05.117349 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.117313    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-workload-certs\") pod \"223aa62e-8850-4abe-9961-4ddbb8df9131\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") "
Sep 15 12:16:05.117349 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.117341    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-credential-socket\") pod \"223aa62e-8850-4abe-9961-4ddbb8df9131\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") "
Sep 15 12:16:05.117621 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.117383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t4pd\" (UniqueName: \"kubernetes.io/projected/223aa62e-8850-4abe-9961-4ddbb8df9131-kube-api-access-6t4pd\") pod \"223aa62e-8850-4abe-9961-4ddbb8df9131\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") "
Sep 15 12:16:05.117621 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.117538    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-podinfo\") pod \"223aa62e-8850-4abe-9961-4ddbb8df9131\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") "
Sep 15 12:16:05.117621 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.117593    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-token\") pod \"223aa62e-8850-4abe-9961-4ddbb8df9131\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") "
Sep 15 12:16:05.117621 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.117598    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "223aa62e-8850-4abe-9961-4ddbb8df9131" (UID: "223aa62e-8850-4abe-9961-4ddbb8df9131"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:05.117819 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.117638    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223aa62e-8850-4abe-9961-4ddbb8df9131-istiod-ca-cert\") pod \"223aa62e-8850-4abe-9961-4ddbb8df9131\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") "
Sep 15 12:16:05.117819 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.117670    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-workload-socket\") pod \"223aa62e-8850-4abe-9961-4ddbb8df9131\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") "
Sep 15 12:16:05.117819 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.117736    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "223aa62e-8850-4abe-9961-4ddbb8df9131" (UID: "223aa62e-8850-4abe-9961-4ddbb8df9131"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:05.117991 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.117896    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-data\") pod \"223aa62e-8850-4abe-9961-4ddbb8df9131\" (UID: \"223aa62e-8850-4abe-9961-4ddbb8df9131\") "
Sep 15 12:16:05.117991 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.117941    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "223aa62e-8850-4abe-9961-4ddbb8df9131" (UID: "223aa62e-8850-4abe-9961-4ddbb8df9131"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:05.118095 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.118038    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223aa62e-8850-4abe-9961-4ddbb8df9131-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "223aa62e-8850-4abe-9961-4ddbb8df9131" (UID: "223aa62e-8850-4abe-9961-4ddbb8df9131"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:05.118268 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.118242    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-data" (OuterVolumeSpecName: "istio-data") pod "223aa62e-8850-4abe-9961-4ddbb8df9131" (UID: "223aa62e-8850-4abe-9961-4ddbb8df9131"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:05.118373 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.118250    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223aa62e-8850-4abe-9961-4ddbb8df9131-istiod-ca-cert\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:05.118373 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.118297    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-workload-socket\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:05.118373 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.118316    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-workload-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:05.118373 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.118332    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-credential-socket\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:05.119646 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.119620    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "223aa62e-8850-4abe-9961-4ddbb8df9131" (UID: "223aa62e-8850-4abe-9961-4ddbb8df9131"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:05.119951 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.119933    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-token" (OuterVolumeSpecName: "istio-token") pod "223aa62e-8850-4abe-9961-4ddbb8df9131" (UID: "223aa62e-8850-4abe-9961-4ddbb8df9131"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:05.120075 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.120056    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "223aa62e-8850-4abe-9961-4ddbb8df9131" (UID: "223aa62e-8850-4abe-9961-4ddbb8df9131"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:16:05.120075 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.120061    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223aa62e-8850-4abe-9961-4ddbb8df9131-kube-api-access-6t4pd" (OuterVolumeSpecName: "kube-api-access-6t4pd") pod "223aa62e-8850-4abe-9961-4ddbb8df9131" (UID: "223aa62e-8850-4abe-9961-4ddbb8df9131"). InnerVolumeSpecName "kube-api-access-6t4pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:05.219190 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.219156    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-podinfo\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:05.219190 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.219182    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-token\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:05.219190 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.219191    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-data\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:05.219190 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.219199    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223aa62e-8850-4abe-9961-4ddbb8df9131-istio-envoy\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:05.219432 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.219209    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t4pd\" (UniqueName: \"kubernetes.io/projected/223aa62e-8850-4abe-9961-4ddbb8df9131-kube-api-access-6t4pd\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:05.325479 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.325444    2580 generic.go:356] "Generic (PLEG): container finished" podID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerID="ffa5a999865025e08f0fc37159451f36ce7d097856ff16e2afc589de91a9caba" exitCode=0
Sep 15 12:16:05.325715 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.325488    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"
Sep 15 12:16:05.325715 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.325522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp" event={"ID":"223aa62e-8850-4abe-9961-4ddbb8df9131","Type":"ContainerDied","Data":"ffa5a999865025e08f0fc37159451f36ce7d097856ff16e2afc589de91a9caba"}
Sep 15 12:16:05.325715 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.325568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp" event={"ID":"223aa62e-8850-4abe-9961-4ddbb8df9131","Type":"ContainerDied","Data":"40e162f9f61d4ea9c950ca94a094665539cdc83d18546245ba0bb2e02c23f816"}
Sep 15 12:16:05.325715 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.325593    2580 scope.go:117] "RemoveContainer" containerID="ffa5a999865025e08f0fc37159451f36ce7d097856ff16e2afc589de91a9caba"
Sep 15 12:16:05.335399 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.335378    2580 scope.go:117] "RemoveContainer" containerID="7d65bc924864dfa4639221b37f2d1369c85608416ecdf816bfbfe7d7c4955a93"
Sep 15 12:16:05.343385 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.343367    2580 scope.go:117] "RemoveContainer" containerID="ffa5a999865025e08f0fc37159451f36ce7d097856ff16e2afc589de91a9caba"
Sep 15 12:16:05.343650 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:16:05.343634    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa5a999865025e08f0fc37159451f36ce7d097856ff16e2afc589de91a9caba\": container with ID starting with ffa5a999865025e08f0fc37159451f36ce7d097856ff16e2afc589de91a9caba not found: ID does not exist" containerID="ffa5a999865025e08f0fc37159451f36ce7d097856ff16e2afc589de91a9caba"
Sep 15 12:16:05.343692 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.343658    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa5a999865025e08f0fc37159451f36ce7d097856ff16e2afc589de91a9caba"} err="failed to get container status \"ffa5a999865025e08f0fc37159451f36ce7d097856ff16e2afc589de91a9caba\": rpc error: code = NotFound desc = could not find container \"ffa5a999865025e08f0fc37159451f36ce7d097856ff16e2afc589de91a9caba\": container with ID starting with ffa5a999865025e08f0fc37159451f36ce7d097856ff16e2afc589de91a9caba not found: ID does not exist"
Sep 15 12:16:05.343692 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.343675    2580 scope.go:117] "RemoveContainer" containerID="7d65bc924864dfa4639221b37f2d1369c85608416ecdf816bfbfe7d7c4955a93"
Sep 15 12:16:05.343924 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:16:05.343907    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d65bc924864dfa4639221b37f2d1369c85608416ecdf816bfbfe7d7c4955a93\": container with ID starting with 7d65bc924864dfa4639221b37f2d1369c85608416ecdf816bfbfe7d7c4955a93 not found: ID does not exist" containerID="7d65bc924864dfa4639221b37f2d1369c85608416ecdf816bfbfe7d7c4955a93"
Sep 15 12:16:05.343968 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.343929    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d65bc924864dfa4639221b37f2d1369c85608416ecdf816bfbfe7d7c4955a93"} err="failed to get container status \"7d65bc924864dfa4639221b37f2d1369c85608416ecdf816bfbfe7d7c4955a93\": rpc error: code = NotFound desc = could not find container \"7d65bc924864dfa4639221b37f2d1369c85608416ecdf816bfbfe7d7c4955a93\": container with ID starting with 7d65bc924864dfa4639221b37f2d1369c85608416ecdf816bfbfe7d7c4955a93 not found: ID does not exist"
Sep 15 12:16:05.352587 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.352566    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"]
Sep 15 12:16:05.359027 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.359004    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7139fb-openshift-default-c64dfcf74-fvwtp"]
Sep 15 12:16:05.731930 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:05.731898    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="223aa62e-8850-4abe-9961-4ddbb8df9131" path="/var/lib/kubelet/pods/223aa62e-8850-4abe-9961-4ddbb8df9131/volumes"
Sep 15 12:16:11.172419 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.172383    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"]
Sep 15 12:16:11.172914 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.172607    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh" podUID="40741ce9-1889-4d94-b5aa-0f6aeae67156" containerName="payload-processing" containerID="cri-o://d1da0853b5df537e7b8d1ed4f1276384f3627d9bc89c582b113b2206aa1a9b95" gracePeriod=30
Sep 15 12:16:11.184034 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.184007    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"]
Sep 15 12:16:11.184231 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.184209    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9" podUID="e7d253f8-9fa5-4c64-8e6f-692bb1da76da" containerName="payload-pre-processing" containerID="cri-o://14f5eb30d06fe924697a98c14c8b1127612aae6c6276dc141d8bbada1d2f35e1" gracePeriod=30
Sep 15 12:16:11.362507 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.362474    2580 generic.go:356] "Generic (PLEG): container finished" podID="e7d253f8-9fa5-4c64-8e6f-692bb1da76da" containerID="14f5eb30d06fe924697a98c14c8b1127612aae6c6276dc141d8bbada1d2f35e1" exitCode=0
Sep 15 12:16:11.362671 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.362560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9" event={"ID":"e7d253f8-9fa5-4c64-8e6f-692bb1da76da","Type":"ContainerDied","Data":"14f5eb30d06fe924697a98c14c8b1127612aae6c6276dc141d8bbada1d2f35e1"}
Sep 15 12:16:11.364629 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.364596    2580 generic.go:356] "Generic (PLEG): container finished" podID="40741ce9-1889-4d94-b5aa-0f6aeae67156" containerID="d1da0853b5df537e7b8d1ed4f1276384f3627d9bc89c582b113b2206aa1a9b95" exitCode=0
Sep 15 12:16:11.364629 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.364637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh" event={"ID":"40741ce9-1889-4d94-b5aa-0f6aeae67156","Type":"ContainerDied","Data":"d1da0853b5df537e7b8d1ed4f1276384f3627d9bc89c582b113b2206aa1a9b95"}
Sep 15 12:16:11.460141 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.460119    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"
Sep 15 12:16:11.463255 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.463234    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"
Sep 15 12:16:11.581664 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.581635    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9cwf\" (UniqueName: \"kubernetes.io/projected/e7d253f8-9fa5-4c64-8e6f-692bb1da76da-kube-api-access-k9cwf\") pod \"e7d253f8-9fa5-4c64-8e6f-692bb1da76da\" (UID: \"e7d253f8-9fa5-4c64-8e6f-692bb1da76da\") "
Sep 15 12:16:11.581819 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.581744    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40741ce9-1889-4d94-b5aa-0f6aeae67156-plugins-config-volume\") pod \"40741ce9-1889-4d94-b5aa-0f6aeae67156\" (UID: \"40741ce9-1889-4d94-b5aa-0f6aeae67156\") "
Sep 15 12:16:11.581819 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.581763    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d253f8-9fa5-4c64-8e6f-692bb1da76da-plugins-config-volume\") pod \"e7d253f8-9fa5-4c64-8e6f-692bb1da76da\" (UID: \"e7d253f8-9fa5-4c64-8e6f-692bb1da76da\") "
Sep 15 12:16:11.581819 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.581792    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-425hr\" (UniqueName: \"kubernetes.io/projected/40741ce9-1889-4d94-b5aa-0f6aeae67156-kube-api-access-425hr\") pod \"40741ce9-1889-4d94-b5aa-0f6aeae67156\" (UID: \"40741ce9-1889-4d94-b5aa-0f6aeae67156\") "
Sep 15 12:16:11.582167 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.582143    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40741ce9-1889-4d94-b5aa-0f6aeae67156-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40741ce9-1889-4d94-b5aa-0f6aeae67156" (UID: "40741ce9-1889-4d94-b5aa-0f6aeae67156"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:11.582240 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.582159    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d253f8-9fa5-4c64-8e6f-692bb1da76da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7d253f8-9fa5-4c64-8e6f-692bb1da76da" (UID: "e7d253f8-9fa5-4c64-8e6f-692bb1da76da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:11.583926 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.583899    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40741ce9-1889-4d94-b5aa-0f6aeae67156-kube-api-access-425hr" (OuterVolumeSpecName: "kube-api-access-425hr") pod "40741ce9-1889-4d94-b5aa-0f6aeae67156" (UID: "40741ce9-1889-4d94-b5aa-0f6aeae67156"). InnerVolumeSpecName "kube-api-access-425hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:11.584065 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.584048    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d253f8-9fa5-4c64-8e6f-692bb1da76da-kube-api-access-k9cwf" (OuterVolumeSpecName: "kube-api-access-k9cwf") pod "e7d253f8-9fa5-4c64-8e6f-692bb1da76da" (UID: "e7d253f8-9fa5-4c64-8e6f-692bb1da76da"). InnerVolumeSpecName "kube-api-access-k9cwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:11.619341 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.619310    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz"]
Sep 15 12:16:11.626236 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.626213    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74aca3-5mfrz"]
Sep 15 12:16:11.683347 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.683285    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40741ce9-1889-4d94-b5aa-0f6aeae67156-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:11.683347 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.683308    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d253f8-9fa5-4c64-8e6f-692bb1da76da-plugins-config-volume\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:11.683347 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.683319    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-425hr\" (UniqueName: \"kubernetes.io/projected/40741ce9-1889-4d94-b5aa-0f6aeae67156-kube-api-access-425hr\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:11.683347 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.683327    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9cwf\" (UniqueName: \"kubernetes.io/projected/e7d253f8-9fa5-4c64-8e6f-692bb1da76da-kube-api-access-k9cwf\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:11.730589 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:11.730561    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a55ebfca-f406-4115-adfe-a967162d4abf" path="/var/lib/kubelet/pods/a55ebfca-f406-4115-adfe-a967162d4abf/volumes"
Sep 15 12:16:12.371715 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:12.371676    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"
Sep 15 12:16:12.372225 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:12.371683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9" event={"ID":"e7d253f8-9fa5-4c64-8e6f-692bb1da76da","Type":"ContainerDied","Data":"1e1f96c2c0da95aac700d0828abaa54b8f493b17f312e2cc07ac2a5db6bad95f"}
Sep 15 12:16:12.372225 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:12.371834    2580 scope.go:117] "RemoveContainer" containerID="14f5eb30d06fe924697a98c14c8b1127612aae6c6276dc141d8bbada1d2f35e1"
Sep 15 12:16:12.373333 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:12.373318    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"
Sep 15 12:16:12.373401 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:12.373347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh" event={"ID":"40741ce9-1889-4d94-b5aa-0f6aeae67156","Type":"ContainerDied","Data":"026a6351c41d43d4e1bc4324a62124b877c9cdb7129c17ec0f08b68f7a58ddc4"}
Sep 15 12:16:12.381693 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:12.381670    2580 scope.go:117] "RemoveContainer" containerID="d1da0853b5df537e7b8d1ed4f1276384f3627d9bc89c582b113b2206aa1a9b95"
Sep 15 12:16:12.399971 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:12.399250    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"]
Sep 15 12:16:12.417671 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:12.417638    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-74aca3-579c5c5c89-zwdzh"]
Sep 15 12:16:12.498939 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:12.498849    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"]
Sep 15 12:16:12.501109 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:12.501088    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-74aca3-67db9cdbdb-ml2t9"]
Sep 15 12:16:13.730733 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:13.730690    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40741ce9-1889-4d94-b5aa-0f6aeae67156" path="/var/lib/kubelet/pods/40741ce9-1889-4d94-b5aa-0f6aeae67156/volumes"
Sep 15 12:16:13.731116 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:13.731100    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7d253f8-9fa5-4c64-8e6f-692bb1da76da" path="/var/lib/kubelet/pods/e7d253f8-9fa5-4c64-8e6f-692bb1da76da/volumes"
Sep 15 12:16:19.840721 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:19.840688    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"]
Sep 15 12:16:19.841149 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:19.840952    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b" podUID="1625e331-b139-4d49-91f5-804068870be2" containerName="istio-proxy" containerID="cri-o://f48ac6f28c39cc5fd3c69b76880091a71500f7d307f672ab6570bf30f43863d5" gracePeriod=30
Sep 15 12:16:20.184459 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.184423    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 12:16:20.184636 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.184503    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b" podUID="1625e331-b139-4d49-91f5-804068870be2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 12:16:20.215567 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.215529    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh"]
Sep 15 12:16:20.215986 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.215967    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerName="istio-proxy"
Sep 15 12:16:20.216095 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.215987    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerName="istio-proxy"
Sep 15 12:16:20.216095 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.216002    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a55ebfca-f406-4115-adfe-a967162d4abf" containerName="revoke-keys"
Sep 15 12:16:20.216095 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.216009    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a55ebfca-f406-4115-adfe-a967162d4abf" containerName="revoke-keys"
Sep 15 12:16:20.216095 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.216020    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerName="istio-proxy"
Sep 15 12:16:20.216095 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.216028    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerName="istio-proxy"
Sep 15 12:16:20.216095 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.216037    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d253f8-9fa5-4c64-8e6f-692bb1da76da" containerName="payload-pre-processing"
Sep 15 12:16:20.216095 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.216045    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d253f8-9fa5-4c64-8e6f-692bb1da76da" containerName="payload-pre-processing"
Sep 15 12:16:20.216095 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.216057    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40741ce9-1889-4d94-b5aa-0f6aeae67156" containerName="payload-processing"
Sep 15 12:16:20.216095 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.216065    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40741ce9-1889-4d94-b5aa-0f6aeae67156" containerName="payload-processing"
Sep 15 12:16:20.216525 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.216161    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a55ebfca-f406-4115-adfe-a967162d4abf" containerName="revoke-keys"
Sep 15 12:16:20.216525 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.216174    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40741ce9-1889-4d94-b5aa-0f6aeae67156" containerName="payload-processing"
Sep 15 12:16:20.216525 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.216188    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerName="istio-proxy"
Sep 15 12:16:20.216525 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.216197    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d253f8-9fa5-4c64-8e6f-692bb1da76da" containerName="payload-pre-processing"
Sep 15 12:16:20.216525 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.216206    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="223aa62e-8850-4abe-9961-4ddbb8df9131" containerName="istio-proxy"
Sep 15 12:16:20.220829 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.220808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh"
Sep 15 12:16:20.225014 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.224985    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:16:20.225120 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.225060    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:16:20.230916 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.230894    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh"]
Sep 15 12:16:20.258042 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.258016    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff20b00-07f6-46b5-99c4-27dd9835cd1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh\" (UID: \"fff20b00-07f6-46b5-99c4-27dd9835cd1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh"
Sep 15 12:16:20.358743 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.358705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff20b00-07f6-46b5-99c4-27dd9835cd1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh\" (UID: \"fff20b00-07f6-46b5-99c4-27dd9835cd1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh"
Sep 15 12:16:20.359318 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.359299    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff20b00-07f6-46b5-99c4-27dd9835cd1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh\" (UID: \"fff20b00-07f6-46b5-99c4-27dd9835cd1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh"
Sep 15 12:16:20.532655 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.532572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh"
Sep 15 12:16:20.810203 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:20.810176    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh"]
Sep 15 12:16:21.429141 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:21.429102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh" event={"ID":"fff20b00-07f6-46b5-99c4-27dd9835cd1a","Type":"ContainerStarted","Data":"a15282e1aa616a127cbfb2f8cdca6ee8c24c3cc5495d15cbb5699321263d93f7"}
Sep 15 12:16:21.429527 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:21.429165    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh" event={"ID":"fff20b00-07f6-46b5-99c4-27dd9835cd1a","Type":"ContainerStarted","Data":"fabdc9f6e86c28b7057d81086b1ee5569cc0d7035776b43c8011f2eca711541a"}
Sep 15 12:16:21.449046 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:21.448991    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh" podStartSLOduration=1.4489749 podStartE2EDuration="1.4489749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:16:21.447524603 +0000 UTC m=+2270.228095982" watchObservedRunningTime="2026-09-15 12:16:21.4489749 +0000 UTC m=+2270.229546288"
Sep 15 12:16:22.435893 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:22.435832    2580 generic.go:356] "Generic (PLEG): container finished" podID="fff20b00-07f6-46b5-99c4-27dd9835cd1a" containerID="a15282e1aa616a127cbfb2f8cdca6ee8c24c3cc5495d15cbb5699321263d93f7" exitCode=0
Sep 15 12:16:22.436347 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:22.435915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh" event={"ID":"fff20b00-07f6-46b5-99c4-27dd9835cd1a","Type":"ContainerDied","Data":"a15282e1aa616a127cbfb2f8cdca6ee8c24c3cc5495d15cbb5699321263d93f7"}
Sep 15 12:16:23.568341 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:23.568315    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh"
Sep 15 12:16:23.589269 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:23.589243    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff20b00-07f6-46b5-99c4-27dd9835cd1a-maas-api-service-ca\") pod \"fff20b00-07f6-46b5-99c4-27dd9835cd1a\" (UID: \"fff20b00-07f6-46b5-99c4-27dd9835cd1a\") "
Sep 15 12:16:23.589691 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:23.589658    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff20b00-07f6-46b5-99c4-27dd9835cd1a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fff20b00-07f6-46b5-99c4-27dd9835cd1a" (UID: "fff20b00-07f6-46b5-99c4-27dd9835cd1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:23.690236 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:23.690205    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff20b00-07f6-46b5-99c4-27dd9835cd1a-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:24.447848 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:24.447809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh" event={"ID":"fff20b00-07f6-46b5-99c4-27dd9835cd1a","Type":"ContainerDied","Data":"fabdc9f6e86c28b7057d81086b1ee5569cc0d7035776b43c8011f2eca711541a"}
Sep 15 12:16:24.447848 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:24.447830    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh"
Sep 15 12:16:24.448136 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:24.447865    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fabdc9f6e86c28b7057d81086b1ee5569cc0d7035776b43c8011f2eca711541a"
Sep 15 12:16:25.093296 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.093272    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:16:25.102906 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.102882    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p8mn\" (UniqueName: \"kubernetes.io/projected/1625e331-b139-4d49-91f5-804068870be2-kube-api-access-2p8mn\") pod \"1625e331-b139-4d49-91f5-804068870be2\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") "
Sep 15 12:16:25.103012 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.102945    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1625e331-b139-4d49-91f5-804068870be2-istio-token\") pod \"1625e331-b139-4d49-91f5-804068870be2\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") "
Sep 15 12:16:25.104751 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.104721    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1625e331-b139-4d49-91f5-804068870be2-kube-api-access-2p8mn" (OuterVolumeSpecName: "kube-api-access-2p8mn") pod "1625e331-b139-4d49-91f5-804068870be2" (UID: "1625e331-b139-4d49-91f5-804068870be2"). InnerVolumeSpecName "kube-api-access-2p8mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:25.104850 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.104787    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1625e331-b139-4d49-91f5-804068870be2-istio-token" (OuterVolumeSpecName: "istio-token") pod "1625e331-b139-4d49-91f5-804068870be2" (UID: "1625e331-b139-4d49-91f5-804068870be2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:25.204332 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204302    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-workload-certs\") pod \"1625e331-b139-4d49-91f5-804068870be2\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") "
Sep 15 12:16:25.204505 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204345    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-workload-socket\") pod \"1625e331-b139-4d49-91f5-804068870be2\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") "
Sep 15 12:16:25.204505 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204379    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-istio-envoy\") pod \"1625e331-b139-4d49-91f5-804068870be2\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") "
Sep 15 12:16:25.204505 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204402    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1625e331-b139-4d49-91f5-804068870be2-istiod-ca-cert\") pod \"1625e331-b139-4d49-91f5-804068870be2\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") "
Sep 15 12:16:25.204505 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204425    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-istio-data\") pod \"1625e331-b139-4d49-91f5-804068870be2\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") "
Sep 15 12:16:25.204505 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204456    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1625e331-b139-4d49-91f5-804068870be2-istio-podinfo\") pod \"1625e331-b139-4d49-91f5-804068870be2\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") "
Sep 15 12:16:25.204505 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204479    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-credential-socket\") pod \"1625e331-b139-4d49-91f5-804068870be2\" (UID: \"1625e331-b139-4d49-91f5-804068870be2\") "
Sep 15 12:16:25.204819 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204561    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1625e331-b139-4d49-91f5-804068870be2" (UID: "1625e331-b139-4d49-91f5-804068870be2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:25.204819 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204618    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1625e331-b139-4d49-91f5-804068870be2" (UID: "1625e331-b139-4d49-91f5-804068870be2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:25.204819 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204740    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-istio-data" (OuterVolumeSpecName: "istio-data") pod "1625e331-b139-4d49-91f5-804068870be2" (UID: "1625e331-b139-4d49-91f5-804068870be2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:25.205003 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204829    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1625e331-b139-4d49-91f5-804068870be2-istio-token\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:25.205003 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204840    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-istio-data\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:25.205003 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204849    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p8mn\" (UniqueName: \"kubernetes.io/projected/1625e331-b139-4d49-91f5-804068870be2-kube-api-access-2p8mn\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:25.205003 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204858    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-workload-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:25.205003 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204866    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1625e331-b139-4d49-91f5-804068870be2" (UID: "1625e331-b139-4d49-91f5-804068870be2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:25.205003 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204921    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1625e331-b139-4d49-91f5-804068870be2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1625e331-b139-4d49-91f5-804068870be2" (UID: "1625e331-b139-4d49-91f5-804068870be2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:25.205003 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.204867    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-workload-socket\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:25.206628 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.206602    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1625e331-b139-4d49-91f5-804068870be2" (UID: "1625e331-b139-4d49-91f5-804068870be2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:25.206755 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.206634    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1625e331-b139-4d49-91f5-804068870be2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1625e331-b139-4d49-91f5-804068870be2" (UID: "1625e331-b139-4d49-91f5-804068870be2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:16:25.305975 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.305945    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-istio-envoy\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:25.305975 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.305969    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1625e331-b139-4d49-91f5-804068870be2-istiod-ca-cert\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:25.305975 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.305979    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1625e331-b139-4d49-91f5-804068870be2-istio-podinfo\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:25.306183 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.305986    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1625e331-b139-4d49-91f5-804068870be2-credential-socket\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:25.456430 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.456400    2580 generic.go:356] "Generic (PLEG): container finished" podID="1625e331-b139-4d49-91f5-804068870be2" containerID="f48ac6f28c39cc5fd3c69b76880091a71500f7d307f672ab6570bf30f43863d5" exitCode=0
Sep 15 12:16:25.456582 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.456444    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"
Sep 15 12:16:25.456582 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.456484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b" event={"ID":"1625e331-b139-4d49-91f5-804068870be2","Type":"ContainerDied","Data":"f48ac6f28c39cc5fd3c69b76880091a71500f7d307f672ab6570bf30f43863d5"}
Sep 15 12:16:25.456582 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.456537    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b" event={"ID":"1625e331-b139-4d49-91f5-804068870be2","Type":"ContainerDied","Data":"933aecd8f3020a46de82242b89ef1c4c8bc351441bee61658e40c0af6d0a2286"}
Sep 15 12:16:25.456582 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.456559    2580 scope.go:117] "RemoveContainer" containerID="f48ac6f28c39cc5fd3c69b76880091a71500f7d307f672ab6570bf30f43863d5"
Sep 15 12:16:25.466550 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.466534    2580 scope.go:117] "RemoveContainer" containerID="6f482cc84f6737d6c90ddc43bccb88d1a18ea8c9ac3a7ce3e8491f75f9ffc40e"
Sep 15 12:16:25.474760 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.474744    2580 scope.go:117] "RemoveContainer" containerID="f48ac6f28c39cc5fd3c69b76880091a71500f7d307f672ab6570bf30f43863d5"
Sep 15 12:16:25.475048 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:16:25.475025    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48ac6f28c39cc5fd3c69b76880091a71500f7d307f672ab6570bf30f43863d5\": container with ID starting with f48ac6f28c39cc5fd3c69b76880091a71500f7d307f672ab6570bf30f43863d5 not found: ID does not exist" containerID="f48ac6f28c39cc5fd3c69b76880091a71500f7d307f672ab6570bf30f43863d5"
Sep 15 12:16:25.475106 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.475054    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48ac6f28c39cc5fd3c69b76880091a71500f7d307f672ab6570bf30f43863d5"} err="failed to get container status \"f48ac6f28c39cc5fd3c69b76880091a71500f7d307f672ab6570bf30f43863d5\": rpc error: code = NotFound desc = could not find container \"f48ac6f28c39cc5fd3c69b76880091a71500f7d307f672ab6570bf30f43863d5\": container with ID starting with f48ac6f28c39cc5fd3c69b76880091a71500f7d307f672ab6570bf30f43863d5 not found: ID does not exist"
Sep 15 12:16:25.475106 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.475069    2580 scope.go:117] "RemoveContainer" containerID="6f482cc84f6737d6c90ddc43bccb88d1a18ea8c9ac3a7ce3e8491f75f9ffc40e"
Sep 15 12:16:25.475311 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:16:25.475292    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f482cc84f6737d6c90ddc43bccb88d1a18ea8c9ac3a7ce3e8491f75f9ffc40e\": container with ID starting with 6f482cc84f6737d6c90ddc43bccb88d1a18ea8c9ac3a7ce3e8491f75f9ffc40e not found: ID does not exist" containerID="6f482cc84f6737d6c90ddc43bccb88d1a18ea8c9ac3a7ce3e8491f75f9ffc40e"
Sep 15 12:16:25.475355 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.475316    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f482cc84f6737d6c90ddc43bccb88d1a18ea8c9ac3a7ce3e8491f75f9ffc40e"} err="failed to get container status \"6f482cc84f6737d6c90ddc43bccb88d1a18ea8c9ac3a7ce3e8491f75f9ffc40e\": rpc error: code = NotFound desc = could not find container \"6f482cc84f6737d6c90ddc43bccb88d1a18ea8c9ac3a7ce3e8491f75f9ffc40e\": container with ID starting with 6f482cc84f6737d6c90ddc43bccb88d1a18ea8c9ac3a7ce3e8491f75f9ffc40e not found: ID does not exist"
Sep 15 12:16:25.486261 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.486240    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"]
Sep 15 12:16:25.488492 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.488471    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-74aca3-openshift-default-84f65574f8-5hm2b"]
Sep 15 12:16:25.730780 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:25.730696    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1625e331-b139-4d49-91f5-804068870be2" path="/var/lib/kubelet/pods/1625e331-b139-4d49-91f5-804068870be2/volumes"
Sep 15 12:16:31.174396 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:31.174360    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b3936-29824575-zt4mq"]
Sep 15 12:16:31.179172 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:31.179144    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2b3936-29824575-zt4mq"]
Sep 15 12:16:31.611952 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:31.611863    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh"]
Sep 15 12:16:31.617649 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:31.617620    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2b3936-9vkwh"]
Sep 15 12:16:31.732446 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:31.732416    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4932a671-5d5e-4d37-91ae-9583064091e9" path="/var/lib/kubelet/pods/4932a671-5d5e-4d37-91ae-9583064091e9/volumes"
Sep 15 12:16:31.732729 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:31.732716    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fff20b00-07f6-46b5-99c4-27dd9835cd1a" path="/var/lib/kubelet/pods/fff20b00-07f6-46b5-99c4-27dd9835cd1a/volumes"
Sep 15 12:16:42.127550 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.127513    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj"]
Sep 15 12:16:42.128074 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.128042    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1625e331-b139-4d49-91f5-804068870be2" containerName="istio-proxy"
Sep 15 12:16:42.128074 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.128064    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1625e331-b139-4d49-91f5-804068870be2" containerName="istio-proxy"
Sep 15 12:16:42.128208 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.128095    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff20b00-07f6-46b5-99c4-27dd9835cd1a" containerName="revoke-keys"
Sep 15 12:16:42.128208 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.128105    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff20b00-07f6-46b5-99c4-27dd9835cd1a" containerName="revoke-keys"
Sep 15 12:16:42.128208 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.128123    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1625e331-b139-4d49-91f5-804068870be2" containerName="istio-proxy"
Sep 15 12:16:42.128208 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.128132    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1625e331-b139-4d49-91f5-804068870be2" containerName="istio-proxy"
Sep 15 12:16:42.128412 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.128209    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1625e331-b139-4d49-91f5-804068870be2" containerName="istio-proxy"
Sep 15 12:16:42.128412 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.128224    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1625e331-b139-4d49-91f5-804068870be2" containerName="istio-proxy"
Sep 15 12:16:42.128412 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.128236    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fff20b00-07f6-46b5-99c4-27dd9835cd1a" containerName="revoke-keys"
Sep 15 12:16:42.131136 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.131114    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj"
Sep 15 12:16:42.140782 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.140759    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:16:42.142948 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.142930    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfdnn\""
Sep 15 12:16:42.146802 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.146767    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj"]
Sep 15 12:16:42.251631 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.251589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj\" (UID: \"e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj"
Sep 15 12:16:42.352673 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.352639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj\" (UID: \"e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj"
Sep 15 12:16:42.353264 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.353244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj\" (UID: \"e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj"
Sep 15 12:16:42.440688 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.440659    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj"
Sep 15 12:16:42.725613 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:42.725587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj"]
Sep 15 12:16:42.727800 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:16:42.727769    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26ebbbd_8fa8_45e2_9cdf_13b3f4da3729.slice/crio-a34f09d68ab7b5dcfddaf2e6382eb6b565ad8f435c857cc43cec18da0c1e30d8 WatchSource:0}: Error finding container a34f09d68ab7b5dcfddaf2e6382eb6b565ad8f435c857cc43cec18da0c1e30d8: Status 404 returned error can't find the container with id a34f09d68ab7b5dcfddaf2e6382eb6b565ad8f435c857cc43cec18da0c1e30d8
Sep 15 12:16:43.559927 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:43.559865    2580 generic.go:356] "Generic (PLEG): container finished" podID="e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729" containerID="f9ae9ada637c7b7eb7cecf9a9572622b3f6395af2739f309ebf22183d096c173" exitCode=0
Sep 15 12:16:43.560287 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:43.559949    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj" event={"ID":"e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729","Type":"ContainerDied","Data":"f9ae9ada637c7b7eb7cecf9a9572622b3f6395af2739f309ebf22183d096c173"}
Sep 15 12:16:43.560287 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:43.560004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj" event={"ID":"e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729","Type":"ContainerStarted","Data":"a34f09d68ab7b5dcfddaf2e6382eb6b565ad8f435c857cc43cec18da0c1e30d8"}
Sep 15 12:16:44.690039 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:44.690018    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj"
Sep 15 12:16:44.775564 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:44.775536    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729-maas-api-service-ca\") pod \"e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729\" (UID: \"e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729\") "
Sep 15 12:16:44.775929 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:44.775908    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729" (UID: "e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:44.876852 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:44.876781    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729-maas-api-service-ca\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:16:45.571791 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:45.571762    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj"
Sep 15 12:16:45.571791 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:45.571773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj" event={"ID":"e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729","Type":"ContainerDied","Data":"a34f09d68ab7b5dcfddaf2e6382eb6b565ad8f435c857cc43cec18da0c1e30d8"}
Sep 15 12:16:45.572030 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:45.571819    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a34f09d68ab7b5dcfddaf2e6382eb6b565ad8f435c857cc43cec18da0c1e30d8"
Sep 15 12:16:52.891909 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:52.891847    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32bbbb-29824575-sxlkr"]
Sep 15 12:16:52.893939 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:52.893914    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32bbbb-29824575-sxlkr"]
Sep 15 12:16:53.259410 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:53.259374    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj"]
Sep 15 12:16:53.265396 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:53.265366    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32bbbb-dclmj"]
Sep 15 12:16:53.730449 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:53.730417    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b23e5f5-9958-40e5-82e4-707bc968dbd2" path="/var/lib/kubelet/pods/1b23e5f5-9958-40e5-82e4-707bc968dbd2/volumes"
Sep 15 12:16:53.730735 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:16:53.730715    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729" path="/var/lib/kubelet/pods/e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729/volumes"
Sep 15 12:20:32.426151 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:20:32.426063    2580 scope.go:117] "RemoveContainer" containerID="35dc07f35411367769fc5aa6bad40e46f6c09aaf3a87242f4944f5ef9df1a306"
Sep 15 12:21:17.572278 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.572239    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"]
Sep 15 12:21:17.572738 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.572564    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729" containerName="revoke-keys"
Sep 15 12:21:17.572738 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.572575    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729" containerName="revoke-keys"
Sep 15 12:21:17.572738 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.572638    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e26ebbbd-8fa8-45e2-9cdf-13b3f4da3729" containerName="revoke-keys"
Sep 15 12:21:17.575653 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.575636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.581457 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.581430    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0b26c14d-kserve-self-signed-certs\""
Sep 15 12:21:17.591754 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.591726    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"]
Sep 15 12:21:17.705303 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.705270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-model-cache\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.705303 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.705308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-home\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.705550 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.705380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8rhq\" (UniqueName: \"kubernetes.io/projected/d8130ac8-d8db-493a-b451-c09c720de8d8-kube-api-access-c8rhq\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.705550 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.705440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-tmp-dir\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.705550 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.705477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8130ac8-d8db-493a-b451-c09c720de8d8-tls-certs\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.705700 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.705566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-dshm\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.806766 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.806730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-dshm\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.806984 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.806788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-model-cache\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.806984 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.806816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-home\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.806984 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.806842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8rhq\" (UniqueName: \"kubernetes.io/projected/d8130ac8-d8db-493a-b451-c09c720de8d8-kube-api-access-c8rhq\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.806984 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.806892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-tmp-dir\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.806984 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.806944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8130ac8-d8db-493a-b451-c09c720de8d8-tls-certs\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.807299 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.807231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-home\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.807359 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.807300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-model-cache\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.807559 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.807538    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-tmp-dir\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.809148 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.809121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-dshm\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.809389 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.809372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8130ac8-d8db-493a-b451-c09c720de8d8-tls-certs\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.818640 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.818615    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8rhq\" (UniqueName: \"kubernetes.io/projected/d8130ac8-d8db-493a-b451-c09c720de8d8-kube-api-access-c8rhq\") pod \"e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") " pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:17.884356 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:17.884285    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:18.161571 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:18.161544    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"]
Sep 15 12:21:18.163027 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:21:18.163001    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8130ac8_d8db_493a_b451_c09c720de8d8.slice/crio-12fcd62d008f3e9c8ab1597c316031edc56a3084612d10aa0d3ed3a1e57c35d5 WatchSource:0}: Error finding container 12fcd62d008f3e9c8ab1597c316031edc56a3084612d10aa0d3ed3a1e57c35d5: Status 404 returned error can't find the container with id 12fcd62d008f3e9c8ab1597c316031edc56a3084612d10aa0d3ed3a1e57c35d5
Sep 15 12:21:19.036848 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:19.036807    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls" event={"ID":"d8130ac8-d8db-493a-b451-c09c720de8d8","Type":"ContainerStarted","Data":"3cc98d61258cf03133d8a0e16e71452bb920bc8b60e9ea6a6a35f8f9553dd763"}
Sep 15 12:21:19.037251 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:19.036860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls" event={"ID":"d8130ac8-d8db-493a-b451-c09c720de8d8","Type":"ContainerStarted","Data":"12fcd62d008f3e9c8ab1597c316031edc56a3084612d10aa0d3ed3a1e57c35d5"}
Sep 15 12:21:19.055582 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:19.055534    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls" podStartSLOduration=2.05551914 podStartE2EDuration="2.05551914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:21:19.055131117 +0000 UTC m=+2567.835702506" watchObservedRunningTime="2026-09-15 12:21:19.05551914 +0000 UTC m=+2567.836090527"
Sep 15 12:21:27.885404 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:27.885358    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:27.885792 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:27.885433    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:27.897596 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:27.897576    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:28.096534 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:28.096503    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:21:29.485049 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.485006    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"]
Sep 15 12:21:29.488544 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.488522    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.492002 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.491984    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0b26c14d-kserve-self-signed-certs\""
Sep 15 12:21:29.508249 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.508224    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"]
Sep 15 12:21:29.613104 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.613074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-tmp-dir\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.613254 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.613136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-dshm\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.613254 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.613166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2l6p\" (UniqueName: \"kubernetes.io/projected/58be9fb6-3737-4817-babb-eab16c358440-kube-api-access-m2l6p\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.613254 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.613187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-home\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.613373 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.613287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58be9fb6-3737-4817-babb-eab16c358440-tls-certs\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.613373 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.613311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-model-cache\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.714184 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.714156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2l6p\" (UniqueName: \"kubernetes.io/projected/58be9fb6-3737-4817-babb-eab16c358440-kube-api-access-m2l6p\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.714351 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.714191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-home\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.714351 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.714233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58be9fb6-3737-4817-babb-eab16c358440-tls-certs\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.714351 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.714248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-model-cache\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.714351 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.714280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-tmp-dir\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.714351 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.714329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-dshm\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.714696 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.714671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-home\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.714779 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.714710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-model-cache\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.714779 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.714764    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-tmp-dir\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.716610 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.716584    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-dshm\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.716710 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.716674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58be9fb6-3737-4817-babb-eab16c358440-tls-certs\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.736126 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.736077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2l6p\" (UniqueName: \"kubernetes.io/projected/58be9fb6-3737-4817-babb-eab16c358440-kube-api-access-m2l6p\") pod \"e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") " pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:29.798126 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:29.798097    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:30.093618 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:30.093590    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"]
Sep 15 12:21:30.095655 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:21:30.095622    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58be9fb6_3737_4817_babb_eab16c358440.slice/crio-1362c1dbbb324f4b9d361751cfb670085245ed678d6710993993365f0a096473 WatchSource:0}: Error finding container 1362c1dbbb324f4b9d361751cfb670085245ed678d6710993993365f0a096473: Status 404 returned error can't find the container with id 1362c1dbbb324f4b9d361751cfb670085245ed678d6710993993365f0a096473
Sep 15 12:21:31.103160 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:31.103120    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx" event={"ID":"58be9fb6-3737-4817-babb-eab16c358440","Type":"ContainerStarted","Data":"c031cad022f31b20b7189141d35587ce8676e4eebcf23b8ad36d1d8f7b0876f1"}
Sep 15 12:21:31.103580 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:31.103169    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx" event={"ID":"58be9fb6-3737-4817-babb-eab16c358440","Type":"ContainerStarted","Data":"1362c1dbbb324f4b9d361751cfb670085245ed678d6710993993365f0a096473"}
Sep 15 12:21:31.123924 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:31.123863    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx" podStartSLOduration=2.123847224 podStartE2EDuration="2.123847224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:21:31.12214715 +0000 UTC m=+2579.902718537" watchObservedRunningTime="2026-09-15 12:21:31.123847224 +0000 UTC m=+2579.904418613"
Sep 15 12:21:32.446273 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:32.446240    2580 scope.go:117] "RemoveContainer" containerID="f3a2bf16f7a4cb6c3c3d4efcc2b57e90ca17080914c9e6274baba3dea24000bd"
Sep 15 12:21:32.454637 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:32.454618    2580 scope.go:117] "RemoveContainer" containerID="90e16a258565588c07e1eab9796e66ee83ea5d24b6c6855c1daff3a99b3d7753"
Sep 15 12:21:32.462632 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:32.462612    2580 scope.go:117] "RemoveContainer" containerID="08219b90d724fd4d35bfd3cc8d4a66a2209234afc0240bd30a6fa541e2ddf54a"
Sep 15 12:21:39.798470 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:39.798434    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:39.798925 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:39.798486    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:39.811117 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:39.811096    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:21:40.162241 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:21:40.162212    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:22:17.584321 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:17.584287    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"]
Sep 15 12:22:19.598289 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:19.598258    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"]
Sep 15 12:22:32.489613 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.489577    2580 scope.go:117] "RemoveContainer" containerID="a11c795e6c16eb75b15e6678a2c08c94dd24755f1050709785fcfd595ccc0762"
Sep 15 12:22:32.497814 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.497794    2580 scope.go:117] "RemoveContainer" containerID="a15282e1aa616a127cbfb2f8cdca6ee8c24c3cc5495d15cbb5699321263d93f7"
Sep 15 12:22:32.602685 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.602651    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls" podUID="d8130ac8-d8db-493a-b451-c09c720de8d8" containerName="main" containerID="cri-o://3cc98d61258cf03133d8a0e16e71452bb920bc8b60e9ea6a6a35f8f9553dd763" gracePeriod=45
Sep 15 12:22:32.845305 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.845281    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:22:32.873580 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.873556    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8130ac8-d8db-493a-b451-c09c720de8d8-tls-certs\") pod \"d8130ac8-d8db-493a-b451-c09c720de8d8\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") "
Sep 15 12:22:32.873709 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.873595    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-home\") pod \"d8130ac8-d8db-493a-b451-c09c720de8d8\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") "
Sep 15 12:22:32.873709 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.873642    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-tmp-dir\") pod \"d8130ac8-d8db-493a-b451-c09c720de8d8\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") "
Sep 15 12:22:32.873709 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.873660    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8rhq\" (UniqueName: \"kubernetes.io/projected/d8130ac8-d8db-493a-b451-c09c720de8d8-kube-api-access-c8rhq\") pod \"d8130ac8-d8db-493a-b451-c09c720de8d8\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") "
Sep 15 12:22:32.873709 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.873682    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-model-cache\") pod \"d8130ac8-d8db-493a-b451-c09c720de8d8\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") "
Sep 15 12:22:32.873947 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.873768    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-dshm\") pod \"d8130ac8-d8db-493a-b451-c09c720de8d8\" (UID: \"d8130ac8-d8db-493a-b451-c09c720de8d8\") "
Sep 15 12:22:32.873947 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.873926    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8130ac8-d8db-493a-b451-c09c720de8d8" (UID: "d8130ac8-d8db-493a-b451-c09c720de8d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:32.874071 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.874010    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8130ac8-d8db-493a-b451-c09c720de8d8" (UID: "d8130ac8-d8db-493a-b451-c09c720de8d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:32.874128 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.874114    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:22:32.874179 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.874136    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:22:32.874179 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.874156    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-home" (OuterVolumeSpecName: "home") pod "d8130ac8-d8db-493a-b451-c09c720de8d8" (UID: "d8130ac8-d8db-493a-b451-c09c720de8d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:32.875783 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.875756    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8130ac8-d8db-493a-b451-c09c720de8d8-kube-api-access-c8rhq" (OuterVolumeSpecName: "kube-api-access-c8rhq") pod "d8130ac8-d8db-493a-b451-c09c720de8d8" (UID: "d8130ac8-d8db-493a-b451-c09c720de8d8"). InnerVolumeSpecName "kube-api-access-c8rhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:32.875921 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.875785    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8130ac8-d8db-493a-b451-c09c720de8d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8130ac8-d8db-493a-b451-c09c720de8d8" (UID: "d8130ac8-d8db-493a-b451-c09c720de8d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:22:32.876160 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.876140    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-dshm" (OuterVolumeSpecName: "dshm") pod "d8130ac8-d8db-493a-b451-c09c720de8d8" (UID: "d8130ac8-d8db-493a-b451-c09c720de8d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:32.975214 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.975183    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:22:32.975214 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.975207    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8130ac8-d8db-493a-b451-c09c720de8d8-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:22:32.975214 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.975216    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8130ac8-d8db-493a-b451-c09c720de8d8-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:22:32.975421 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:32.975226    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8rhq\" (UniqueName: \"kubernetes.io/projected/d8130ac8-d8db-493a-b451-c09c720de8d8-kube-api-access-c8rhq\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:22:33.443693 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:33.443654    2580 generic.go:356] "Generic (PLEG): container finished" podID="d8130ac8-d8db-493a-b451-c09c720de8d8" containerID="3cc98d61258cf03133d8a0e16e71452bb920bc8b60e9ea6a6a35f8f9553dd763" exitCode=0
Sep 15 12:22:33.443843 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:33.443700    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"
Sep 15 12:22:33.443843 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:33.443746    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls" event={"ID":"d8130ac8-d8db-493a-b451-c09c720de8d8","Type":"ContainerDied","Data":"3cc98d61258cf03133d8a0e16e71452bb920bc8b60e9ea6a6a35f8f9553dd763"}
Sep 15 12:22:33.443843 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:33.443792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls" event={"ID":"d8130ac8-d8db-493a-b451-c09c720de8d8","Type":"ContainerDied","Data":"12fcd62d008f3e9c8ab1597c316031edc56a3084612d10aa0d3ed3a1e57c35d5"}
Sep 15 12:22:33.443843 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:33.443809    2580 scope.go:117] "RemoveContainer" containerID="3cc98d61258cf03133d8a0e16e71452bb920bc8b60e9ea6a6a35f8f9553dd763"
Sep 15 12:22:33.453444 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:33.453424    2580 scope.go:117] "RemoveContainer" containerID="3cc98d61258cf03133d8a0e16e71452bb920bc8b60e9ea6a6a35f8f9553dd763"
Sep 15 12:22:33.453680 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:22:33.453660    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc98d61258cf03133d8a0e16e71452bb920bc8b60e9ea6a6a35f8f9553dd763\": container with ID starting with 3cc98d61258cf03133d8a0e16e71452bb920bc8b60e9ea6a6a35f8f9553dd763 not found: ID does not exist" containerID="3cc98d61258cf03133d8a0e16e71452bb920bc8b60e9ea6a6a35f8f9553dd763"
Sep 15 12:22:33.453760 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:33.453687    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc98d61258cf03133d8a0e16e71452bb920bc8b60e9ea6a6a35f8f9553dd763"} err="failed to get container status \"3cc98d61258cf03133d8a0e16e71452bb920bc8b60e9ea6a6a35f8f9553dd763\": rpc error: code = NotFound desc = could not find container \"3cc98d61258cf03133d8a0e16e71452bb920bc8b60e9ea6a6a35f8f9553dd763\": container with ID starting with 3cc98d61258cf03133d8a0e16e71452bb920bc8b60e9ea6a6a35f8f9553dd763 not found: ID does not exist"
Sep 15 12:22:33.467971 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:33.467946    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"]
Sep 15 12:22:33.474237 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:33.474213    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0b26c14d-kserve-cffd5565f-n4mls"]
Sep 15 12:22:33.731342 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:33.731260    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8130ac8-d8db-493a-b451-c09c720de8d8" path="/var/lib/kubelet/pods/d8130ac8-d8db-493a-b451-c09c720de8d8/volumes"
Sep 15 12:22:34.617065 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.617027    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx" podUID="58be9fb6-3737-4817-babb-eab16c358440" containerName="main" containerID="cri-o://c031cad022f31b20b7189141d35587ce8676e4eebcf23b8ad36d1d8f7b0876f1" gracePeriod=45
Sep 15 12:22:34.868954 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.868898    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:22:34.892285 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.892259    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-dshm\") pod \"58be9fb6-3737-4817-babb-eab16c358440\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") "
Sep 15 12:22:34.892401 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.892338    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-model-cache\") pod \"58be9fb6-3737-4817-babb-eab16c358440\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") "
Sep 15 12:22:34.892401 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.892367    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-tmp-dir\") pod \"58be9fb6-3737-4817-babb-eab16c358440\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") "
Sep 15 12:22:34.892520 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.892403    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58be9fb6-3737-4817-babb-eab16c358440-tls-certs\") pod \"58be9fb6-3737-4817-babb-eab16c358440\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") "
Sep 15 12:22:34.892520 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.892433    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2l6p\" (UniqueName: \"kubernetes.io/projected/58be9fb6-3737-4817-babb-eab16c358440-kube-api-access-m2l6p\") pod \"58be9fb6-3737-4817-babb-eab16c358440\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") "
Sep 15 12:22:34.892520 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.892457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-home\") pod \"58be9fb6-3737-4817-babb-eab16c358440\" (UID: \"58be9fb6-3737-4817-babb-eab16c358440\") "
Sep 15 12:22:34.892675 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.892633    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-model-cache" (OuterVolumeSpecName: "model-cache") pod "58be9fb6-3737-4817-babb-eab16c358440" (UID: "58be9fb6-3737-4817-babb-eab16c358440"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:34.892675 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.892661    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58be9fb6-3737-4817-babb-eab16c358440" (UID: "58be9fb6-3737-4817-babb-eab16c358440"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:34.892838 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.892814    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-home" (OuterVolumeSpecName: "home") pod "58be9fb6-3737-4817-babb-eab16c358440" (UID: "58be9fb6-3737-4817-babb-eab16c358440"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:34.892838 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.892820    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-model-cache\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:22:34.893055 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.892854    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-tmp-dir\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:22:34.894378 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.894358    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-dshm" (OuterVolumeSpecName: "dshm") pod "58be9fb6-3737-4817-babb-eab16c358440" (UID: "58be9fb6-3737-4817-babb-eab16c358440"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:34.894616 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.894591    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58be9fb6-3737-4817-babb-eab16c358440-kube-api-access-m2l6p" (OuterVolumeSpecName: "kube-api-access-m2l6p") pod "58be9fb6-3737-4817-babb-eab16c358440" (UID: "58be9fb6-3737-4817-babb-eab16c358440"). InnerVolumeSpecName "kube-api-access-m2l6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:34.894814 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.894785    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58be9fb6-3737-4817-babb-eab16c358440-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58be9fb6-3737-4817-babb-eab16c358440" (UID: "58be9fb6-3737-4817-babb-eab16c358440"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:22:34.993560 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.993531    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58be9fb6-3737-4817-babb-eab16c358440-tls-certs\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:22:34.993709 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.993560    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2l6p\" (UniqueName: \"kubernetes.io/projected/58be9fb6-3737-4817-babb-eab16c358440-kube-api-access-m2l6p\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:22:34.993709 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.993580    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-home\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:22:34.993709 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:34.993589    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be9fb6-3737-4817-babb-eab16c358440-dshm\") on node \"ip-10-0-143-211.ec2.internal\" DevicePath \"\""
Sep 15 12:22:35.457435 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:35.457401    2580 generic.go:356] "Generic (PLEG): container finished" podID="58be9fb6-3737-4817-babb-eab16c358440" containerID="c031cad022f31b20b7189141d35587ce8676e4eebcf23b8ad36d1d8f7b0876f1" exitCode=0
Sep 15 12:22:35.457617 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:35.457451    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"
Sep 15 12:22:35.457617 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:35.457458    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx" event={"ID":"58be9fb6-3737-4817-babb-eab16c358440","Type":"ContainerDied","Data":"c031cad022f31b20b7189141d35587ce8676e4eebcf23b8ad36d1d8f7b0876f1"}
Sep 15 12:22:35.457617 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:35.457552    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx" event={"ID":"58be9fb6-3737-4817-babb-eab16c358440","Type":"ContainerDied","Data":"1362c1dbbb324f4b9d361751cfb670085245ed678d6710993993365f0a096473"}
Sep 15 12:22:35.457617 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:35.457570    2580 scope.go:117] "RemoveContainer" containerID="c031cad022f31b20b7189141d35587ce8676e4eebcf23b8ad36d1d8f7b0876f1"
Sep 15 12:22:35.466687 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:35.466668    2580 scope.go:117] "RemoveContainer" containerID="c031cad022f31b20b7189141d35587ce8676e4eebcf23b8ad36d1d8f7b0876f1"
Sep 15 12:22:35.466994 ip-10-0-143-211 kubenswrapper[2580]: E0915 12:22:35.466973    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c031cad022f31b20b7189141d35587ce8676e4eebcf23b8ad36d1d8f7b0876f1\": container with ID starting with c031cad022f31b20b7189141d35587ce8676e4eebcf23b8ad36d1d8f7b0876f1 not found: ID does not exist" containerID="c031cad022f31b20b7189141d35587ce8676e4eebcf23b8ad36d1d8f7b0876f1"
Sep 15 12:22:35.467069 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:35.467000    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c031cad022f31b20b7189141d35587ce8676e4eebcf23b8ad36d1d8f7b0876f1"} err="failed to get container status \"c031cad022f31b20b7189141d35587ce8676e4eebcf23b8ad36d1d8f7b0876f1\": rpc error: code = NotFound desc = could not find container \"c031cad022f31b20b7189141d35587ce8676e4eebcf23b8ad36d1d8f7b0876f1\": container with ID starting with c031cad022f31b20b7189141d35587ce8676e4eebcf23b8ad36d1d8f7b0876f1 not found: ID does not exist"
Sep 15 12:22:35.484752 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:35.484722    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"]
Sep 15 12:22:35.486777 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:35.486757    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0b26c14d-kserve-799dcd8d55-w52qx"]
Sep 15 12:22:35.730044 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:22:35.729963    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58be9fb6-3737-4817-babb-eab16c358440" path="/var/lib/kubelet/pods/58be9fb6-3737-4817-babb-eab16c358440/volumes"
Sep 15 12:23:15.867182 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:15.867145    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mxqg4_7008f68e-d6ba-4717-a8da-9976436b8b89/server/0.log"
Sep 15 12:23:15.982891 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:15.982848    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-97ktw_36595136-4851-48d3-9beb-28b17fda498c/manager/0.log"
Sep 15 12:23:16.106426 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:16.106395    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66bc7bc78c-2f7rq_de62f8dd-efac-420e-ae3b-e67de86faad7/copy-manifests/0.log"
Sep 15 12:23:16.126383 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:16.126310    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66bc7bc78c-2f7rq_de62f8dd-efac-420e-ae3b-e67de86faad7/manager/0.log"
Sep 15 12:23:17.066139 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:17.066102    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29824575-djw7l_372c3ac5-4781-48bd-9faa-5b43e5b2f02d/cleanup/0.log"
Sep 15 12:23:19.056647 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:19.056616    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p_6930cf41-976c-4b06-a522-9fc7f633e8e0/istio-proxy/0.log"
Sep 15 12:23:19.399582 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:19.399499    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-df8dfc748-6hsvp_6dba8ee1-dd2e-429c-9585-3411f89d407b/kube-auth-proxy/0.log"
Sep 15 12:23:19.756409 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:19.756381    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-j556w_b9653379-1bc2-4fb8-9c12-9a92ca8eb998/payload-processing/0.log"
Sep 15 12:23:20.454310 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:20.454276    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-8cg5r_5c2ff8be-58a6-4c6a-bd64-41cfe26ae38c/main/0.log"
Sep 15 12:23:20.793093 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:20.793010    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-tvszs_f0a5e0c0-71ba-43aa-8272-91158e33853e/main/0.log"
Sep 15 12:23:20.907244 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:20.907204    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-fh5vc_e5c187a7-7ffa-42a2-bf62-f9d437620759/main/0.log"
Sep 15 12:23:32.211616 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:32.211521    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6tnl8_ec648b53-d8f0-43aa-82d3-5ae9212975d6/global-pull-secret-syncer/0.log"
Sep 15 12:23:32.512818 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:32.512725    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pdj4g_3ed568cb-25cc-452a-a618-ef7b524de14a/konnectivity-agent/0.log"
Sep 15 12:23:32.522130 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:32.522104    2580 scope.go:117] "RemoveContainer" containerID="f9ae9ada637c7b7eb7cecf9a9572622b3f6395af2739f309ebf22183d096c173"
Sep 15 12:23:32.626599 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:32.626575    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-211.ec2.internal_acc80c84a62ea6b15d405529c0cb63ed/haproxy/0.log"
Sep 15 12:23:40.115780 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:40.115700    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qrj8n_ade6c623-622d-4e47-892d-3bb15ccec90b/node-exporter/0.log"
Sep 15 12:23:40.146837 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:40.146797    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qrj8n_ade6c623-622d-4e47-892d-3bb15ccec90b/kube-rbac-proxy/0.log"
Sep 15 12:23:40.179921 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:40.179897    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qrj8n_ade6c623-622d-4e47-892d-3bb15ccec90b/init-textfile/0.log"
Sep 15 12:23:42.184863 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:42.184834    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-c6p6s_77711f0c-75a3-40c6-be1d-a0aaa5ca2ad2/networking-console-plugin/0.log"
Sep 15 12:23:43.459889 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:43.459843    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74d9fc7f7-65kmw_e387e513-3eda-48c5-9ef8-dd565a73b8aa/console/0.log"
Sep 15 12:23:44.777903 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.777856    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"]
Sep 15 12:23:44.778441 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.778418    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58be9fb6-3737-4817-babb-eab16c358440" containerName="main"
Sep 15 12:23:44.778530 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.778445    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="58be9fb6-3737-4817-babb-eab16c358440" containerName="main"
Sep 15 12:23:44.778530 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.778464    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8130ac8-d8db-493a-b451-c09c720de8d8" containerName="main"
Sep 15 12:23:44.778530 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.778472    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8130ac8-d8db-493a-b451-c09c720de8d8" containerName="main"
Sep 15 12:23:44.778681 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.778581    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8130ac8-d8db-493a-b451-c09c720de8d8" containerName="main"
Sep 15 12:23:44.778681 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.778596    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="58be9fb6-3737-4817-babb-eab16c358440" containerName="main"
Sep 15 12:23:44.781418 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.781396    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:44.784685 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.784665    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q4mgh\"/\"default-dockercfg-gg5x2\""
Sep 15 12:23:44.784775 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.784665    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q4mgh\"/\"openshift-service-ca.crt\""
Sep 15 12:23:44.785288 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.785273    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q4mgh\"/\"kube-root-ca.crt\""
Sep 15 12:23:44.795529 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.795507    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"]
Sep 15 12:23:44.903147 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.903110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7452dc2c-2a61-463e-b537-70e1d57fb52c-sys\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:44.903147 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.903146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6278\" (UniqueName: \"kubernetes.io/projected/7452dc2c-2a61-463e-b537-70e1d57fb52c-kube-api-access-c6278\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:44.903391 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.903167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7452dc2c-2a61-463e-b537-70e1d57fb52c-proc\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:44.903391 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.903249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7452dc2c-2a61-463e-b537-70e1d57fb52c-podres\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:44.903498 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:44.903405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7452dc2c-2a61-463e-b537-70e1d57fb52c-lib-modules\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:45.004775 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.004743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7452dc2c-2a61-463e-b537-70e1d57fb52c-lib-modules\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:45.004939 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.004790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7452dc2c-2a61-463e-b537-70e1d57fb52c-sys\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:45.004939 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.004853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7452dc2c-2a61-463e-b537-70e1d57fb52c-sys\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:45.004939 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.004912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6278\" (UniqueName: \"kubernetes.io/projected/7452dc2c-2a61-463e-b537-70e1d57fb52c-kube-api-access-c6278\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:45.005065 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.004941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7452dc2c-2a61-463e-b537-70e1d57fb52c-proc\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:45.005065 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.004961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7452dc2c-2a61-463e-b537-70e1d57fb52c-podres\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:45.005065 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.004980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7452dc2c-2a61-463e-b537-70e1d57fb52c-lib-modules\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:45.005065 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.005001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7452dc2c-2a61-463e-b537-70e1d57fb52c-proc\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:45.005191 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.005095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7452dc2c-2a61-463e-b537-70e1d57fb52c-podres\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:45.014633 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.014609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6278\" (UniqueName: \"kubernetes.io/projected/7452dc2c-2a61-463e-b537-70e1d57fb52c-kube-api-access-c6278\") pod \"perf-node-gather-daemonset-pwxrs\" (UID: \"7452dc2c-2a61-463e-b537-70e1d57fb52c\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:45.091511 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.091430    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:45.362314 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.362285    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"]
Sep 15 12:23:45.364074 ip-10-0-143-211 kubenswrapper[2580]: W0915 12:23:45.364044    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7452dc2c_2a61_463e_b537_70e1d57fb52c.slice/crio-a3cdb6e4a3ed28711c9985e40a500a8db7c5c13bb2a3e033843e5098dcd45d64 WatchSource:0}: Error finding container a3cdb6e4a3ed28711c9985e40a500a8db7c5c13bb2a3e033843e5098dcd45d64: Status 404 returned error can't find the container with id a3cdb6e4a3ed28711c9985e40a500a8db7c5c13bb2a3e033843e5098dcd45d64
Sep 15 12:23:45.454962 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.454939    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6pcgn_e2ea32c9-131c-42c0-b364-7802d07ff5d8/dns/0.log"
Sep 15 12:23:45.486518 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.486480    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6pcgn_e2ea32c9-131c-42c0-b364-7802d07ff5d8/kube-rbac-proxy/0.log"
Sep 15 12:23:45.710427 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.710399    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wjqmg_5467c994-9999-4b50-880d-538763991a9e/dns-node-resolver/0.log"
Sep 15 12:23:45.816070 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.816039    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs" event={"ID":"7452dc2c-2a61-463e-b537-70e1d57fb52c","Type":"ContainerStarted","Data":"4cc44ca8b8f4fd989fffafe1698fac80ff0d882b43ec92044404df0d7e4b29d3"}
Sep 15 12:23:45.816440 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.816086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs" event={"ID":"7452dc2c-2a61-463e-b537-70e1d57fb52c","Type":"ContainerStarted","Data":"a3cdb6e4a3ed28711c9985e40a500a8db7c5c13bb2a3e033843e5098dcd45d64"}
Sep 15 12:23:45.816440 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.816134    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:45.836282 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:45.836230    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs" podStartSLOduration=1.8362127560000001 podStartE2EDuration="1.836212756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:23:45.834557068 +0000 UTC m=+2714.615128455" watchObservedRunningTime="2026-09-15 12:23:45.836212756 +0000 UTC m=+2714.616784143"
Sep 15 12:23:46.348584 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:46.348550    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8d687b44b-k6v8j_1cd82af1-94cb-452e-93e8-f6178b5fbe8d/registry/0.log"
Sep 15 12:23:46.411949 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:46.411923    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hd8pm_d00a7cd2-9093-4f86-b40b-1b29b8f6d061/node-ca/0.log"
Sep 15 12:23:47.563973 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:47.563944    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p_6930cf41-976c-4b06-a522-9fc7f633e8e0/istio-proxy/0.log"
Sep 15 12:23:48.330016 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:48.329982    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-df8dfc748-6hsvp_6dba8ee1-dd2e-429c-9585-3411f89d407b/kube-auth-proxy/0.log"
Sep 15 12:23:48.461790 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:48.461753    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-j556w_b9653379-1bc2-4fb8-9c12-9a92ca8eb998/payload-processing/0.log"
Sep 15 12:23:49.504207 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:49.504174    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6pvbr_657cdce3-0695-499e-af03-9e946aa5d6d5/serve-healthcheck-canary/0.log"
Sep 15 12:23:50.207510 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:50.207473    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7nvb_2e994d99-39e5-4133-9f60-26839d83ab39/kube-rbac-proxy/0.log"
Sep 15 12:23:50.233484 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:50.233462    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7nvb_2e994d99-39e5-4133-9f60-26839d83ab39/exporter/0.log"
Sep 15 12:23:50.260140 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:50.260117    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7nvb_2e994d99-39e5-4133-9f60-26839d83ab39/extractor/0.log"
Sep 15 12:23:51.835494 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:51.835467    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-pwxrs"
Sep 15 12:23:53.250449 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:53.250418    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p_6930cf41-976c-4b06-a522-9fc7f633e8e0/istio-proxy/0.log"
Sep 15 12:23:53.313770 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:53.313745    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mxqg4_7008f68e-d6ba-4717-a8da-9976436b8b89/server/0.log"
Sep 15 12:23:53.434609 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:53.434550    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-97ktw_36595136-4851-48d3-9beb-28b17fda498c/manager/0.log"
Sep 15 12:23:53.541115 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:53.541037    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66bc7bc78c-2f7rq_de62f8dd-efac-420e-ae3b-e67de86faad7/manager/0.log"
Sep 15 12:23:53.585640 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:53.585620    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66bc7bc78c-2f7rq_de62f8dd-efac-420e-ae3b-e67de86faad7/copy-manifests/0.log"
Sep 15 12:23:53.994478 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:53.994442    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-df8dfc748-6hsvp_6dba8ee1-dd2e-429c-9585-3411f89d407b/kube-auth-proxy/0.log"
Sep 15 12:23:54.092224 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:54.092195    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-j556w_b9653379-1bc2-4fb8-9c12-9a92ca8eb998/payload-processing/0.log"
Sep 15 12:23:55.397372 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:55.397343    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-c47996ddf-ppd9d_8043bd57-fec0-49a0-8889-fe42a0a9c253/manager/0.log"
Sep 15 12:23:55.838619 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:55.838581    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-sfh2p_6930cf41-976c-4b06-a522-9fc7f633e8e0/istio-proxy/0.log"
Sep 15 12:23:56.755586 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:56.755558    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-df8dfc748-6hsvp_6dba8ee1-dd2e-429c-9585-3411f89d407b/kube-auth-proxy/0.log"
Sep 15 12:23:56.845789 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:23:56.845712    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-j556w_b9653379-1bc2-4fb8-9c12-9a92ca8eb998/payload-processing/0.log"
Sep 15 12:24:01.430588 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:24:01.430553    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-v76nb_4c462336-f677-41ac-94cb-a3e83fd57a3e/kube-storage-version-migrator-operator/1.log"
Sep 15 12:24:01.431551 ip-10-0-143-211 kubenswrapper[2580]: I0915 12:24:01.431523    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-v76nb_4c462336-f677-41ac-94cb-a3e83fd57a3e/kube-storage-version-migrator-operator/0.log"