Sep 09 16:08:43.956005 ip-10-0-137-117 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 16:08:44.338527 ip-10-0-137-117 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 16:08:44.338527 ip-10-0-137-117 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 16:08:44.338527 ip-10-0-137-117 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 16:08:44.338527 ip-10-0-137-117 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 16:08:44.338527 ip-10-0-137-117 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 16:08:44.340982 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.340904    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 16:08:44.345014 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.344999    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:08:44.345014 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345015    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345018    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345022    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345025    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345028    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345030    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345036    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345039    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345042    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345045    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345048    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345051    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345053    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345057    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345059    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345062    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345065    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345068    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345070    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345073    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:08:44.345075 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345076    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345079    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345082    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345085    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345088    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345091    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345093    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345096    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345098    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345101    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345103    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345106    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345108    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345111    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345113    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345116    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345118    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345123    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345125    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345128    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:08:44.345578 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345131    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345133    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345136    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345138    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345140    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345143    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345145    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345148    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345150    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345153    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345155    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345157    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345160    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345162    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345167    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345170    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345173    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345175    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345178    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:08:44.346094 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345180    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345185    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345188    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345191    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345193    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345196    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345198    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345201    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345203    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345205    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345208    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345211    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345213    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345216    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345219    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345221    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345223    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345226    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345228    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345231    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:08:44.346562 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345233    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345236    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345238    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345242    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345246    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345249    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345252    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345256    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345259    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345263    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345267    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345271    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345274    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345277    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345280    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345283    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345286    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345289    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345292    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:08:44.347078 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.345998    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346005    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346008    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346011    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346013    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346016    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346020    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346023    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346026    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346028    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346033    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346037    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346040    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346043    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346046    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346049    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346052    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346055    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346058    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:08:44.347537 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346061    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346064    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346067    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346070    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346073    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346076    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346078    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346081    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346084    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346086    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346089    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346091    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346094    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346096    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346099    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346101    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346104    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346107    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346110    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346113    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:08:44.348026 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346116    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346118    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346121    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346124    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346126    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346129    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346132    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346135    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346137    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346141    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346143    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346146    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346148    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346151    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346154    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346157    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346159    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346163    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346167    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346170    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:08:44.348507 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346172    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346175    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346179    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346182    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346184    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346187    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346189    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346192    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346195    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346197    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346199    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346202    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346205    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346208    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346211    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346214    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346217    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346255    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346264    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:08:44.349028 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346268    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346273    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346276    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346279    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346282    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346285    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346288    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346291    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346294    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346297    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346299    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346302    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346305    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346307    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346310    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346313    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346315    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346318    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346321    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346323    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:08:44.349505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346326    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346725    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346730    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346734    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346736    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346739    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346742    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346744    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346747    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346750    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346754    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346757    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346760    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346762    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346765    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346768    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346771    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346773    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346776    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346780    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:08:44.350012 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346784    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346787    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346790    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346793    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346795    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346798    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346800    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346804    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346808    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346811    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346813    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346816    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346818    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346821    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346823    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346826    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346828    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346834    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346837    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346840    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:08:44.350579 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346842    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346845    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346848    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346851    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346854    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346856    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346859    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346861    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346864    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346867    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346869    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346872    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346874    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346877    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346880    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346883    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346885    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346888    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346890    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346893    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:08:44.351175 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346895    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346898    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346900    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346902    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346905    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346908    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346910    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346914    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346917    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346923    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346926    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346930    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346933    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346935    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346938    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346941    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346943    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346946    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346949    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346951    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:08:44.351662 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346953    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346956    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346958    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346960    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346963    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346965    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346968    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346970    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346973    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346975    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346978    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346980    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346983    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346985    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346987    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346990    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346993    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346995    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.346998    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347000    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:08:44.352160 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347068    2576 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347105    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347118    2576 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347127    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347131    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347135    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347139    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347143    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347146    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347151    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347154    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347158    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347160    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347163    2576 flags.go:64] FLAG: --cgroup-root=""
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347166    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347169    2576 flags.go:64] FLAG: --client-ca-file=""
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347172    2576 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347175    2576 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347179    2576 flags.go:64] FLAG: --cluster-domain=""
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347182    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347185    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347188    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347191    2576 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347195    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 16:08:44.352676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347197    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347200    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347204    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347207    2576 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347209    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347212    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347216    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347219    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347222    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347226    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347229    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347232    2576 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347235    2576 flags.go:64] FLAG: --enable-server="true"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347238    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347242    2576 flags.go:64] FLAG: --event-burst="100"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347245    2576 flags.go:64] FLAG: --event-qps="50"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347248    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347251    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347255    2576 flags.go:64] FLAG: --eviction-hard=""
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347259    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347262    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347265    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347268    2576 flags.go:64] FLAG: --eviction-soft=""
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347271    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347273    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 16:08:44.353259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347276    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347279    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347282    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347285    2576 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347287    2576 flags.go:64] FLAG: --feature-gates=""
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347291    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347294    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347297    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347300    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347302    2576 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347305    2576 flags.go:64] FLAG: --help="false"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347308    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347311    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347314    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347316    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347320    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347324    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347327    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347330    2576 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347333    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347335    2576 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347338    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347342    2576 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347344    2576 flags.go:64] FLAG: --kube-reserved=""
Sep 09 16:08:44.353907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347347    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347350    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347353    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347356    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347359    2576 flags.go:64] FLAG: --lock-file=""
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347362    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347365    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347367    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347373    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347376    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347378    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347381    2576 flags.go:64] FLAG: --logging-format="text"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347384    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347387    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347390    2576 flags.go:64] FLAG: --manifest-url=""
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347392    2576 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347397    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347399    2576 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347403    2576 flags.go:64] FLAG: --max-pods="110"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347406    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347409    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347412    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347415    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347418    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347421    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 16:08:44.354525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347424    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347430    2576 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347433    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347436    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347439    2576 flags.go:64] FLAG: --pod-cidr=""
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347442    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347447    2576 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347450    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347453    2576 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347456    2576 flags.go:64] FLAG: --port="10250"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347460    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347463    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00e95b36c13f75469"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347466    2576 flags.go:64] FLAG: --qos-reserved=""
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347469    2576 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347472    2576 flags.go:64] FLAG: --register-node="true"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347475    2576 flags.go:64] FLAG: --register-with-taints=""
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347479    2576 flags.go:64] FLAG: --registry-burst="10"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347481    2576 flags.go:64] FLAG: --registry-qps="5"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347484    2576 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347487    2576 flags.go:64] FLAG: --reserved-memory=""
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347490    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347493    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347496    2576 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347499    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347501    2576 flags.go:64] FLAG: --runonce="false"
Sep 09 16:08:44.355199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347504    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347507    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347510    2576 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347513    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347516    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347519    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347528    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347531    2576 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347534    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347537    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347541    2576 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347544    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347547    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347550    2576 flags.go:64] FLAG: --system-cgroups=""
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347553    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347558    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347561    2576 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347564    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347568    2576 flags.go:64] FLAG: --tls-min-version=""
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347571    2576 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347573    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347576    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347579    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347582    2576 flags.go:64] FLAG: --v="2"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347585    2576 flags.go:64] FLAG: --version="false"
Sep 09 16:08:44.355930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347589    2576 flags.go:64] FLAG: --vmodule=""
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347593    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.347596    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347719    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347722    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347725    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347727    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347730    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347733    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347735    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347738    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347740    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347743    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347748    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347752    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347754    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347757    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347760    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347762    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:08:44.356902 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347765    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347767    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347770    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347772    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347775    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347777    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347780    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347783    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347785    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347788    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347791    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347796    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347798    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347802    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347804    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347807    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347810    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347813    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347815    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347818    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:08:44.357376 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347821    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347823    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347825    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347828    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347830    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347833    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347839    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347843    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347845    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347848    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347851    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347854    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347856    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347859    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347862    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347865    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347867    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347870    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347872    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:08:44.357939 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347875    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347878    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347881    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347883    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347885    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347888    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347891    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347895    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347899    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347903    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347906    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347909    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347912    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347914    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347917    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347919    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347922    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347924    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347926    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347931    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:08:44.358408 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347933    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347936    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347938    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347941    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347943    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347946    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347948    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347951    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347953    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347955    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347958    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347960    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347963    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347965    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347968    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347971    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347973    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347975    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347978    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347981    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:08:44.358968 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347983    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:08:44.359460 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347987    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:08:44.359460 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347990    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:08:44.359460 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.347992    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:08:44.359460 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.348498    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 16:08:44.359460 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.358065    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 16:08:44.359460 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.358081    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 16:08:44.359460 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.358100    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 16:08:44.359460 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.358106    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 16:08:44.359460 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358161    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:08:44.359460 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358167    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:08:44.359460 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358171    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:08:44.359460 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358174    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:08:44.359460 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358177    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:08:44.359460 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358180    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358183    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358185    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358188    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358190    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358193    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358195    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358198    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358200    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358203    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358206    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358209    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358211    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358214    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358216    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358219    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358221    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358224    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358226    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:08:44.359828 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358228    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358231    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358233    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358236    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358238    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358241    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358244    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358246    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358249    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358251    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358254    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358256    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358259    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358261    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358264    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358266    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358269    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358271    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358273    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358276    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:08:44.360321 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358278    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358281    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358284    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358287    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358289    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358292    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358294    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358296    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358300    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358304    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358307    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358309    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358312    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358314    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358317    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358319    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358321    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358325    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358329    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:08:44.360817 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358332    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358335    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358338    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358341    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358344    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358348    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358350    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358353    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358356    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358359    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358362    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358364    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358367    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358369    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358372    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358374    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358377    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358379    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358382    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358384    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:08:44.361279 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358386    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358389    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358391    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358394    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358396    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358399    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358401    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358405    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358408    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358411    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358414    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358417    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358420    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358423    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358425    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:08:44.361779 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358428    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.358434    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358564    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358570    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358574    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358577    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358580    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358583    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358585    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358588    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358591    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358593    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358596    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358598    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358601    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:08:44.362142 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358604    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358606    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358609    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358611    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358614    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358617    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358619    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358622    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358624    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358627    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358629    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358632    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358635    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358637    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358640    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358642    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358645    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358647    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358649    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358652    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:08:44.362518 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358655    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358658    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358661    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358665    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358668    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358671    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358674    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358677    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358679    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358682    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358684    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358687    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358690    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358720    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358723    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358726    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358728    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358731    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358733    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:08:44.363050 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358736    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358738    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358740    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358743    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358746    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358750    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358753    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358757    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358761    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358765    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358768    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358771    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358774    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358777    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358779    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358782    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358784    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358787    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358790    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:08:44.363504 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358792    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358794    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358797    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358799    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358802    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358804    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358807    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358810    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358812    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358815    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358817    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358820    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358822    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358824    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358827    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358829    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358832    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358834    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358838    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:08:44.363970 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358840    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:08:44.364452 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358842    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:08:44.364452 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358845    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:08:44.364452 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358847    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:08:44.364452 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358850    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:08:44.364452 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358853    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:08:44.364452 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358855    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:08:44.364452 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358858    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:08:44.364452 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:44.358860    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:08:44.364452 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.358865    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 16:08:44.364452 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.358986    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 16:08:44.364452 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.360605    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 16:08:44.364452 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.361525    2576 server.go:1019] "Starting client certificate rotation"
Sep 09 16:08:44.364452 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.361628    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 16:08:44.364452 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.361656    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 16:08:44.364815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.363982    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 16:08:44.367003 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.366987    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 16:08:44.376596 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.376577    2576 log.go:25] "Validated CRI v1 runtime API"
Sep 09 16:08:44.381322 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.381305    2576 log.go:25] "Validated CRI v1 image API"
Sep 09 16:08:44.390209 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.390183    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 16:08:44.392852 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.392829    2576 fs.go:135] Filesystem UUIDs: map[6de9412d-0362-44ad-a024-b20c03fff2ac:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8e4deb56-014a-40c4-8352-8ec76bd1ea1d:/dev/nvme0n1p4]
Sep 09 16:08:44.392943 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.392848    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 16:08:44.394524 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.394505    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 16:08:44.398620 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.398512    2576 manager.go:216] Machine: {Timestamp:2026-09-09 16:08:44.397211647 +0000 UTC m=+0.332830065 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099089 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec233cab1411aa25d3acda29655e854e SystemUUID:ec233cab-1411-aa25-d3ac-da29655e854e BootID:31e6f705-a7ab-4934-8562-be1268833837 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e3:ef:85:a7:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e3:ef:85:a7:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:e8:6b:03:1b:f4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 16:08:44.398620 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.398618    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 16:08:44.398765 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.398714    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 16:08:44.403027 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.402794    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 16:08:44.403174 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.403031    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-117.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 16:08:44.404211 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.404199    2576 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 16:08:44.404255 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.404215    2576 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 16:08:44.404255 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.404222    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 16:08:44.404255 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.404234    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 16:08:44.405436 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.405426    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 16:08:44.408261 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.408248    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 16:08:44.408377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.408368    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 16:08:44.408407 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.408401    2576 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 16:08:44.408449 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.408415    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 16:08:44.408449 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.408438    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 16:08:44.408522 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.408452    2576 kubelet.go:398] "Adding apiserver pod source"
Sep 09 16:08:44.408522 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.408464    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 16:08:44.408910 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.408898    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 16:08:44.408948 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.408917    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 16:08:44.411638 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.411621    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 16:08:44.413858 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.413844    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 16:08:44.413918 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.413876    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 16:08:44.414743 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.414732    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 16:08:44.414788 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.414750    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 16:08:44.414788 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.414757    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 16:08:44.414788 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.414762    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 16:08:44.414788 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.414771    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 16:08:44.414788 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.414776    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 16:08:44.414976 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.414794    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 16:08:44.414976 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.414802    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 16:08:44.414976 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.414811    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 16:08:44.414976 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.414820    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 16:08:44.414976 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.414831    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 16:08:44.414976 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.414839    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 16:08:44.414976 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.414898    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p5tsf"
Sep 09 16:08:44.415550 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.415537    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 16:08:44.415550 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.415548    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 16:08:44.419677 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.419659    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-117.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 16:08:44.419770 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.419662    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 16:08:44.419770 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.419668    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 16:08:44.419770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.419730    2576 server.go:1268] "Started kubelet"
Sep 09 16:08:44.419914 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.419829    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 16:08:44.419965 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.419904    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 16:08:44.419965 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.419956    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 16:08:44.420825 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.420806    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p5tsf"
Sep 09 16:08:44.421300 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.421286    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 16:08:44.422139 ip-10-0-137-117 systemd[1]: Started Kubernetes Kubelet.
Sep 09 16:08:44.424886 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.424871    2576 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 16:08:44.427087 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.427069    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 16:08:44.427518 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.426378    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-117.ec2.internal.18d3b2d98bd264e9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-117.ec2.internal,UID:ip-10-0-137-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-117.ec2.internal,},FirstTimestamp:2026-09-09 16:08:44.419687657 +0000 UTC m=+0.355306067,LastTimestamp:2026-09-09 16:08:44.419687657 +0000 UTC m=+0.355306067,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-117.ec2.internal,}"
Sep 09 16:08:44.427622 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.427538    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 16:08:44.427839 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.427816    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 16:08:44.428083 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.428058    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:44.428342 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.428322    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 16:08:44.428424 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.428347    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 16:08:44.428680 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.428347    2576 factory.go:55] Registering systemd factory
Sep 09 16:08:44.428758 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.428686    2576 factory.go:223] Registration of the systemd container factory successfully
Sep 09 16:08:44.428758 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.428670    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 16:08:44.428758 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.428742    2576 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 16:08:44.428758 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.428751    2576 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 16:08:44.428943 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.428886    2576 factory.go:153] Registering CRI-O factory
Sep 09 16:08:44.428943 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.428898    2576 factory.go:223] Registration of the crio container factory successfully
Sep 09 16:08:44.429038 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.428961    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 16:08:44.429038 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.428976    2576 factory.go:103] Registering Raw factory
Sep 09 16:08:44.429038 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.428986    2576 manager.go:1196] Started watching for new ooms in manager
Sep 09 16:08:44.429391 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.429380    2576 manager.go:349] Starting recovery of all containers
Sep 09 16:08:44.435317 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.435145    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:08:44.438104 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.438074    2576 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-117.ec2.internal\" not found" node="ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.443841 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.443826    2576 manager.go:354] Recovery completed
Sep 09 16:08:44.447952 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.447938    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:08:44.450989 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.450974    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:08:44.451042 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.451006    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:08:44.451042 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.451016    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:08:44.451593 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.451578    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 16:08:44.451656 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.451593    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 16:08:44.451656 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.451611    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 16:08:44.451749 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.451595    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:08:44.453194 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.453183    2576 policy_none.go:49] "None policy: Start"
Sep 09 16:08:44.453238 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.453197    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 16:08:44.453238 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.453207    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 16:08:44.454184 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.454171    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:08:44.454222 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.454198    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:08:44.454222 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.454212    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:08:44.454316 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.454305    2576 policy_none.go:47] "Start"
Sep 09 16:08:44.493102 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.493085    2576 manager.go:337] "Starting Device Plugin manager"
Sep 09 16:08:44.493216 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.493121    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 16:08:44.493216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.493132    2576 server.go:87] "Starting device plugin registration server"
Sep 09 16:08:44.493479 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.493465    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 16:08:44.493537 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.493480    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 16:08:44.494278 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.493675    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 16:08:44.494278 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.493774    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 16:08:44.494278 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.493783    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 16:08:44.494278 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.494206    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 16:08:44.494278 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.494257    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:44.545643 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.545598    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 16:08:44.546875 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.546857    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 16:08:44.546959 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.546902    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 16:08:44.546959 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.546925    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 16:08:44.547056 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.546967    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 16:08:44.549306 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.549265    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:08:44.593759 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.593655    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:08:44.594629 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.594613    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:08:44.594727 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.594651    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:08:44.594727 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.594662    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:08:44.594727 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.594686    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.600746 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.600730    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.600831 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.600757    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-117.ec2.internal\": node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:44.615685 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.615666    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:44.648005 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.647984    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-117.ec2.internal"]
Sep 09 16:08:44.648077 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.648045    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:08:44.649602 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.649586    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:08:44.649707 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.649620    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:08:44.649707 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.649630    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:08:44.650879 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.650866    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:08:44.651011 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.650981    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.651104 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.651018    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:08:44.651581 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.651567    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:08:44.651662 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.651598    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:08:44.651662 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.651567    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:08:44.651662 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.651612    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:08:44.651662 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.651628    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:08:44.651662 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.651640    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:08:44.652988 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.652972    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.653099 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.653003    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:08:44.657646 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.657630    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:08:44.657742 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.657657    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:08:44.657742 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.657668    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:08:44.671210 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.671185    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-117.ec2.internal\" not found" node="ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.676893 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.676877    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-117.ec2.internal\" not found" node="ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.716316 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.716296    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:44.730637 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.730616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a79f993c43fd2759c85c24894aebebdd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal\" (UID: \"a79f993c43fd2759c85c24894aebebdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.730757 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.730659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a79f993c43fd2759c85c24894aebebdd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal\" (UID: \"a79f993c43fd2759c85c24894aebebdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.730757 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.730688    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/12a5a478ae082f068e29cf7483267f02-config\") pod \"kube-apiserver-proxy-ip-10-0-137-117.ec2.internal\" (UID: \"12a5a478ae082f068e29cf7483267f02\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.816810 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.816782    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:44.831140 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.831122    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a79f993c43fd2759c85c24894aebebdd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal\" (UID: \"a79f993c43fd2759c85c24894aebebdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.831211 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.831163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a79f993c43fd2759c85c24894aebebdd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal\" (UID: \"a79f993c43fd2759c85c24894aebebdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.831211 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.831193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a79f993c43fd2759c85c24894aebebdd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal\" (UID: \"a79f993c43fd2759c85c24894aebebdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.831285 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.831210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/12a5a478ae082f068e29cf7483267f02-config\") pod \"kube-apiserver-proxy-ip-10-0-137-117.ec2.internal\" (UID: \"12a5a478ae082f068e29cf7483267f02\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.831285 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.831240    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/12a5a478ae082f068e29cf7483267f02-config\") pod \"kube-apiserver-proxy-ip-10-0-137-117.ec2.internal\" (UID: \"12a5a478ae082f068e29cf7483267f02\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.831285 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.831261    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a79f993c43fd2759c85c24894aebebdd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal\" (UID: \"a79f993c43fd2759c85c24894aebebdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.917541 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:44.917481    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:44.974000 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.973969    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:44.980236 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:44.980209    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:45.017977 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:45.017950    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:45.118470 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:45.118424    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:45.219072 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:45.218986    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:45.319478 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:45.319448    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:45.361988 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:45.361948    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 16:08:45.362422 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:45.362107    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 16:08:45.362422 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:45.362154    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 16:08:45.420395 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:45.420357    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:45.425915 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:45.425886    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 16:03:44 +0000 UTC" deadline="2028-02-02 17:26:25.519513568 +0000 UTC"
Sep 09 16:08:45.425915 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:45.425913    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12265h17m40.093603816s"
Sep 09 16:08:45.521048 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:45.520974    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:45.621401 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:45.621377    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:45.722184 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:45.722147    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-117.ec2.internal\" not found"
Sep 09 16:08:45.730489 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:45.730453    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79f993c43fd2759c85c24894aebebdd.slice/crio-34582529689c65da5bbf9e40418eadb6c7d67699de84db10d9ee9731ad4bfbe0 WatchSource:0}: Error finding container 34582529689c65da5bbf9e40418eadb6c7d67699de84db10d9ee9731ad4bfbe0: Status 404 returned error can't find the container with id 34582529689c65da5bbf9e40418eadb6c7d67699de84db10d9ee9731ad4bfbe0
Sep 09 16:08:45.730778 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:45.730751    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a5a478ae082f068e29cf7483267f02.slice/crio-4144e94e0198c8fbcbc9de6f38ba46ed65bbd879ff6ebb2c7b4eb68b1c3e4b8c WatchSource:0}: Error finding container 4144e94e0198c8fbcbc9de6f38ba46ed65bbd879ff6ebb2c7b4eb68b1c3e4b8c: Status 404 returned error can't find the container with id 4144e94e0198c8fbcbc9de6f38ba46ed65bbd879ff6ebb2c7b4eb68b1c3e4b8c
Sep 09 16:08:45.730865 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:45.730839    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:08:45.735923 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:45.735909    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:08:45.748154 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:45.748130    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:08:45.828676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:45.828653    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:45.839784 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:45.839761    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 16:08:45.840850 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:45.840835    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-117.ec2.internal"
Sep 09 16:08:45.847259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:45.847244    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 16:08:46.410142 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.410108    2576 apiserver.go:52] "Watching apiserver"
Sep 09 16:08:46.417221 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.417198    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 16:08:46.417711 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.417668    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal","openshift-multus/network-metrics-daemon-nwfs2","openshift-ovn-kubernetes/ovnkube-node-td8ml","kube-system/global-pull-secret-syncer-psbxz","kube-system/kube-apiserver-proxy-ip-10-0-137-117.ec2.internal","openshift-dns/node-resolver-5fhrb","openshift-image-registry/node-ca-tlk4z","openshift-multus/multus-additional-cni-plugins-ghgq7","openshift-multus/multus-b9zp2","openshift-network-diagnostics/network-check-target-8njz2","openshift-network-operator/iptables-alerter-tm757","kube-system/konnectivity-agent-nrwxd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q","openshift-cluster-node-tuning-operator/tuned-lf495"]
Sep 09 16:08:46.419439 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.419403    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tlk4z"
Sep 09 16:08:46.421363 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.421342    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tm757"
Sep 09 16:08:46.422423 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.422402    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 16:08:46.422516 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.422424    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 16:08:46.422516 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.422456    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-997nh\""
Sep 09 16:08:46.422634 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.422514    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 16:08:46.423501 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.423480    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 16:08:46.423877 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.423856    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 16:08:46.423955 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.423917    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 16:08:46.424032 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.423864    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7gmbr\""
Sep 09 16:08:46.424910 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.424272    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.425428 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.425407    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:46.425533 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:46.425504    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:08:46.426590 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.426570    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 16:08:46.426691 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.426623    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 16:08:46.427586 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.427288    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.428016 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.427996    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 16:08:46.428156 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.428134    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 16:08:46.428237 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.428219    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 16:08:46.428363 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.428346    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 16:08:46.428468 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.428451    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9f7xv\""
Sep 09 16:08:46.428638 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.428621    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 16:08:46.430244 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.430225    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hxbbt\""
Sep 09 16:08:46.430349 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.430335    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5fhrb"
Sep 09 16:08:46.430786 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.430749    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 16:08:46.430786 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.430781    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 16:08:46.432770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.431604    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.432770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.432735    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 16:08:46.433088 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.433070    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 16:08:46.433305 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.433291    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gccmf\""
Sep 09 16:08:46.434130 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.433939    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 16:08:46.435458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.434703    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2gclv\""
Sep 09 16:08:46.435458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.434703    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 16:08:46.435458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.434802    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 16:08:46.435458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.434830    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 16:08:46.435458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.435356    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.438223 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.437850    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:46.438223 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:46.437959    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:08:46.438223 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.437990    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 16:08:46.438223 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.437994    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 16:08:46.439189 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.439168    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 16:08:46.439290 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.439179    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sgk8s\""
Sep 09 16:08:46.440685 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.440590    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:08:46.440685 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:46.440650    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:08:46.440861 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.440786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8264777d-c185-4879-8c20-68855787c674-etc-tuned\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.440861 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.440821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-run-netns\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.440861 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.440843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-log-socket\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.440861 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.440858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-sysctl-conf\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.441084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.440873    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-systemd\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.441084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.440887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96e4e031-e5c2-46a3-a044-0b0d6da36fb1-serviceca\") pod \"node-ca-tlk4z\" (UID: \"96e4e031-e5c2-46a3-a044-0b0d6da36fb1\") " pod="openshift-image-registry/node-ca-tlk4z"
Sep 09 16:08:46.441084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.440902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs5mw\" (UniqueName: \"kubernetes.io/projected/96e4e031-e5c2-46a3-a044-0b0d6da36fb1-kube-api-access-cs5mw\") pod \"node-ca-tlk4z\" (UID: \"96e4e031-e5c2-46a3-a044-0b0d6da36fb1\") " pod="openshift-image-registry/node-ca-tlk4z"
Sep 09 16:08:46.441084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.440916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/279dc324-8320-4f43-886a-f152f85a5e4a-host-slash\") pod \"iptables-alerter-tm757\" (UID: \"279dc324-8320-4f43-886a-f152f85a5e4a\") " pod="openshift-network-operator/iptables-alerter-tm757"
Sep 09 16:08:46.441084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.440932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-kubelet\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.441084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.440959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.441084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.440976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv6hx\" (UniqueName: \"kubernetes.io/projected/31eb8e52-4636-4856-a696-385e9a058d49-kube-api-access-bv6hx\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.441084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441001    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-slash\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.441084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441017    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-var-lib-openvswitch\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.441084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441033    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-run-openvswitch\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.441084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441047    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27913e87-038f-437f-a49e-6e3768968886-dbus\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:46.441084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-modprobe-d\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.441084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441076    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-lib-modules\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441100    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96e4e031-e5c2-46a3-a044-0b0d6da36fb1-host\") pod \"node-ca-tlk4z\" (UID: \"96e4e031-e5c2-46a3-a044-0b0d6da36fb1\") " pod="openshift-image-registry/node-ca-tlk4z"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-run-ovn-kubernetes\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441150    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-var-lib-kubelet\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-host\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441196    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8264777d-c185-4879-8c20-68855787c674-tmp\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6g9k\" (UniqueName: \"kubernetes.io/projected/0e49d49c-6384-4195-8385-374e3552fc45-kube-api-access-q6g9k\") pod \"node-resolver-5fhrb\" (UID: \"0e49d49c-6384-4195-8385-374e3552fc45\") " pod="openshift-dns/node-resolver-5fhrb"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441290    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/279dc324-8320-4f43-886a-f152f85a5e4a-iptables-alerter-script\") pod \"iptables-alerter-tm757\" (UID: \"279dc324-8320-4f43-886a-f152f85a5e4a\") " pod="openshift-network-operator/iptables-alerter-tm757"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvx6w\" (UniqueName: \"kubernetes.io/projected/279dc324-8320-4f43-886a-f152f85a5e4a-kube-api-access-hvx6w\") pod \"iptables-alerter-tm757\" (UID: \"279dc324-8320-4f43-886a-f152f85a5e4a\") " pod="openshift-network-operator/iptables-alerter-tm757"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31eb8e52-4636-4856-a696-385e9a058d49-ovn-node-metrics-cert\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31eb8e52-4636-4856-a696-385e9a058d49-ovnkube-script-lib\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441466    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27913e87-038f-437f-a49e-6e3768968886-kubelet-config\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441494    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-kubernetes\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441521    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-sysctl-d\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-run-systemd\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.441594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-cni-bin\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.442216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-cni-netd\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.442216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31eb8e52-4636-4856-a696-385e9a058d49-env-overrides\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.442216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441691    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-sys\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.442216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441718    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e49d49c-6384-4195-8385-374e3552fc45-hosts-file\") pod \"node-resolver-5fhrb\" (UID: \"0e49d49c-6384-4195-8385-374e3552fc45\") " pod="openshift-dns/node-resolver-5fhrb"
Sep 09 16:08:46.442216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e49d49c-6384-4195-8385-374e3552fc45-tmp-dir\") pod \"node-resolver-5fhrb\" (UID: \"0e49d49c-6384-4195-8385-374e3552fc45\") " pod="openshift-dns/node-resolver-5fhrb"
Sep 09 16:08:46.442216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-systemd-units\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.442216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441920    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-run-ovn\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.442216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.441986    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:46.442216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.442012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-sysconfig\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.442216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.442039    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-run\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.442216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.442064    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z8lm\" (UniqueName: \"kubernetes.io/projected/8264777d-c185-4879-8c20-68855787c674-kube-api-access-8z8lm\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.442216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.442089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-etc-openvswitch\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.442216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.442114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-node-log\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.442216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.442139    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31eb8e52-4636-4856-a696-385e9a058d49-ovnkube-config\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.442878 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.442391    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 16:08:46.442878 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.442610    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.442878 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.442793    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nrwxd"
Sep 09 16:08:46.445054 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.445036    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2l99j\""
Sep 09 16:08:46.445270 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.445248    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 16:08:46.445362 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.445271    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 16:08:46.445512 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.445489    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 16:08:46.445613 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.445576    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 16:08:46.445730 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.445715    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p5sjg\""
Sep 09 16:08:46.464915 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.464894    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rwnvx"
Sep 09 16:08:46.471320 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.471303    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rwnvx"
Sep 09 16:08:46.529964 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.529944    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 16:08:46.542520 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.542498    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.542642 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.542535    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-run-systemd\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.542642 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.542563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-sys\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.542642 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.542582    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e49d49c-6384-4195-8385-374e3552fc45-tmp-dir\") pod \"node-resolver-5fhrb\" (UID: \"0e49d49c-6384-4195-8385-374e3552fc45\") " pod="openshift-dns/node-resolver-5fhrb"
Sep 09 16:08:46.542642 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.542600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-systemd-units\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.542642 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.542616    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:46.542642 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.542629    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-run\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.542642 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.542634    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-run-systemd\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.543017 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.542645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z8lm\" (UniqueName: \"kubernetes.io/projected/8264777d-c185-4879-8c20-68855787c674-kube-api-access-8z8lm\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.543017 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.542713    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-sys\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.543017 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:46.542849    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:08:46.543017 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:46.542940    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret podName:27913e87-038f-437f-a49e-6e3768968886 nodeName:}" failed. No retries permitted until 2026-09-09 16:08:47.042904479 +0000 UTC m=+2.978522895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret") pod "global-pull-secret-syncer-psbxz" (UID: "27913e87-038f-437f-a49e-6e3768968886") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:08:46.543017 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.542985    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-run\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.543280 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543033    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e49d49c-6384-4195-8385-374e3552fc45-tmp-dir\") pod \"node-resolver-5fhrb\" (UID: \"0e49d49c-6384-4195-8385-374e3552fc45\") " pod="openshift-dns/node-resolver-5fhrb"
Sep 09 16:08:46.543280 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543104    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-systemd-units\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.543280 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.542696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/896fdb8e-35d7-4032-8b48-54f32d74e4eb-multus-daemon-config\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.543280 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543146    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-node-log\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.543280 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8264777d-c185-4879-8c20-68855787c674-etc-tuned\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.543280 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-var-lib-cni-bin\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.543280 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543225    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-var-lib-kubelet\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.543280 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7466397e-16a1-4782-89b9-3f2c611e2bd0-cnibin\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.543280 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543277    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-run-netns\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-log-socket\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-sysctl-conf\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543359    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-system-cni-dir\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543385    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543410    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7466397e-16a1-4782-89b9-3f2c611e2bd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-socket-dir\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96e4e031-e5c2-46a3-a044-0b0d6da36fb1-serviceca\") pod \"node-ca-tlk4z\" (UID: \"96e4e031-e5c2-46a3-a044-0b0d6da36fb1\") " pod="openshift-image-registry/node-ca-tlk4z"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543481    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs5mw\" (UniqueName: \"kubernetes.io/projected/96e4e031-e5c2-46a3-a044-0b0d6da36fb1-kube-api-access-cs5mw\") pod \"node-ca-tlk4z\" (UID: \"96e4e031-e5c2-46a3-a044-0b0d6da36fb1\") " pod="openshift-image-registry/node-ca-tlk4z"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/279dc324-8320-4f43-886a-f152f85a5e4a-host-slash\") pod \"iptables-alerter-tm757\" (UID: \"279dc324-8320-4f43-886a-f152f85a5e4a\") " pod="openshift-network-operator/iptables-alerter-tm757"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-kubelet\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv6hx\" (UniqueName: \"kubernetes.io/projected/31eb8e52-4636-4856-a696-385e9a058d49-kube-api-access-bv6hx\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-run-multus-certs\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543611    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543640    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-var-lib-openvswitch\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543651    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-node-log\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.543770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-run-openvswitch\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27913e87-038f-437f-a49e-6e3768968886-dbus\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-lib-modules\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543763    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-run-netns\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6g9k\" (UniqueName: \"kubernetes.io/projected/0e49d49c-6384-4195-8385-374e3552fc45-kube-api-access-q6g9k\") pod \"node-resolver-5fhrb\" (UID: \"0e49d49c-6384-4195-8385-374e3552fc45\") " pod="openshift-dns/node-resolver-5fhrb"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543814    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-log-socket\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543828    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543846    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-var-lib-cni-multus\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543872    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-multus-conf-dir\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96e4e031-e5c2-46a3-a044-0b0d6da36fb1-host\") pod \"node-ca-tlk4z\" (UID: \"96e4e031-e5c2-46a3-a044-0b0d6da36fb1\") " pod="openshift-image-registry/node-ca-tlk4z"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-host\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543932    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-sysctl-conf\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8264777d-c185-4879-8c20-68855787c674-tmp\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.543976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzjmz\" (UniqueName: \"kubernetes.io/projected/03652816-9d0a-45d3-9390-04cc6acc8cee-kube-api-access-mzjmz\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7466397e-16a1-4782-89b9-3f2c611e2bd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544009    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-var-lib-openvswitch\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a-agent-certs\") pod \"konnectivity-agent-nrwxd\" (UID: \"dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a\") " pod="kube-system/konnectivity-agent-nrwxd"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544077    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31eb8e52-4636-4856-a696-385e9a058d49-ovn-node-metrics-cert\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.544678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31eb8e52-4636-4856-a696-385e9a058d49-ovnkube-script-lib\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544145    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvlzp\" (UniqueName: \"kubernetes.io/projected/896fdb8e-35d7-4032-8b48-54f32d74e4eb-kube-api-access-wvlzp\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544170    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcgp6\" (UniqueName: \"kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6\") pod \"network-check-target-8njz2\" (UID: \"af80a467-9aa6-4c3b-81fd-d05dc3f09e75\") " pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544196    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-cni-bin\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-cni-netd\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31eb8e52-4636-4856-a696-385e9a058d49-env-overrides\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e49d49c-6384-4195-8385-374e3552fc45-hosts-file\") pod \"node-resolver-5fhrb\" (UID: \"0e49d49c-6384-4195-8385-374e3552fc45\") " pod="openshift-dns/node-resolver-5fhrb"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-hostroot\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544346    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-run-ovn\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544373    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-sysconfig\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7466397e-16a1-4782-89b9-3f2c611e2bd0-system-cni-dir\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-etc-openvswitch\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544444    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31eb8e52-4636-4856-a696-385e9a058d49-ovnkube-config\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.544527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-multus-cni-dir\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.545037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-run-netns\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.545078    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-etc-kubernetes\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.545595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.545574    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7466397e-16a1-4782-89b9-3f2c611e2bd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.546501 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.545678    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-systemd\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.546501 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.546240    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/279dc324-8320-4f43-886a-f152f85a5e4a-host-slash\") pod \"iptables-alerter-tm757\" (UID: \"279dc324-8320-4f43-886a-f152f85a5e4a\") " pod="openshift-network-operator/iptables-alerter-tm757"
Sep 09 16:08:46.546501 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.546293    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-kubelet\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.546861 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.546823    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31eb8e52-4636-4856-a696-385e9a058d49-ovnkube-config\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.546970 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.546945    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27913e87-038f-437f-a49e-6e3768968886-dbus\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:46.547025 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.546965    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-cni-bin\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.547025 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.546967    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-run-ovn\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.547025 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.547019    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e49d49c-6384-4195-8385-374e3552fc45-hosts-file\") pod \"node-resolver-5fhrb\" (UID: \"0e49d49c-6384-4195-8385-374e3552fc45\") " pod="openshift-dns/node-resolver-5fhrb"
Sep 09 16:08:46.547190 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.547025    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-cni-netd\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.547190 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.547070    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-run-openvswitch\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.547190 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.547129    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-sysconfig\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.547346 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.547222    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-host\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.547346 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.547320    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-lib-modules\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.547621 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.547594    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31eb8e52-4636-4856-a696-385e9a058d49-ovnkube-script-lib\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.547701 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.547627    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96e4e031-e5c2-46a3-a044-0b0d6da36fb1-host\") pod \"node-ca-tlk4z\" (UID: \"96e4e031-e5c2-46a3-a044-0b0d6da36fb1\") " pod="openshift-image-registry/node-ca-tlk4z"
Sep 09 16:08:46.547701 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.547689    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-etc-openvswitch\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.548126 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.548104    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96e4e031-e5c2-46a3-a044-0b0d6da36fb1-serviceca\") pod \"node-ca-tlk4z\" (UID: \"96e4e031-e5c2-46a3-a044-0b0d6da36fb1\") " pod="openshift-image-registry/node-ca-tlk4z"
Sep 09 16:08:46.548226 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.548202    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31eb8e52-4636-4856-a696-385e9a058d49-env-overrides\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.548650 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.548628    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8264777d-c185-4879-8c20-68855787c674-etc-tuned\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.548735 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.548684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-systemd\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.548815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.548754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-cnibin\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.548815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.548793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7466397e-16a1-4782-89b9-3f2c611e2bd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.548927 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.548828    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-registration-dir\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.548927 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.548860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-device-dir\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.548927 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.548893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-slash\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.548927 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.548919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-modprobe-d\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.549140 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.548951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-run-k8s-cni-cncf-io\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.549140 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.548982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-run-ovn-kubernetes\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.549140 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-var-lib-kubelet\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.549140 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549045    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/896fdb8e-35d7-4032-8b48-54f32d74e4eb-cni-binary-copy\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.549140 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-multus-socket-dir-parent\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.549140 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549103    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv99w\" (UniqueName: \"kubernetes.io/projected/7466397e-16a1-4782-89b9-3f2c611e2bd0-kube-api-access-jv99w\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.549140 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a-konnectivity-ca\") pod \"konnectivity-agent-nrwxd\" (UID: \"dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a\") " pod="kube-system/konnectivity-agent-nrwxd"
Sep 09 16:08:46.549558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-sys-fs\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.549558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549188    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5qp\" (UniqueName: \"kubernetes.io/projected/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-kube-api-access-vz5qp\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.549558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/279dc324-8320-4f43-886a-f152f85a5e4a-iptables-alerter-script\") pod \"iptables-alerter-tm757\" (UID: \"279dc324-8320-4f43-886a-f152f85a5e4a\") " pod="openshift-network-operator/iptables-alerter-tm757"
Sep 09 16:08:46.549558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvx6w\" (UniqueName: \"kubernetes.io/projected/279dc324-8320-4f43-886a-f152f85a5e4a-kube-api-access-hvx6w\") pod \"iptables-alerter-tm757\" (UID: \"279dc324-8320-4f43-886a-f152f85a5e4a\") " pod="openshift-network-operator/iptables-alerter-tm757"
Sep 09 16:08:46.549558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27913e87-038f-437f-a49e-6e3768968886-kubelet-config\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:46.549558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-kubernetes\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.549558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-sysctl-d\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.549939 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549580    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-sysctl-d\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.549939 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549678    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-slash\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.549939 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549758    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-modprobe-d\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.549939 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31eb8e52-4636-4856-a696-385e9a058d49-host-run-ovn-kubernetes\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.549939 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.549905    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-var-lib-kubelet\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.550518 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.550492    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/279dc324-8320-4f43-886a-f152f85a5e4a-iptables-alerter-script\") pod \"iptables-alerter-tm757\" (UID: \"279dc324-8320-4f43-886a-f152f85a5e4a\") " pod="openshift-network-operator/iptables-alerter-tm757"
Sep 09 16:08:46.550933 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.550895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27913e87-038f-437f-a49e-6e3768968886-kubelet-config\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:46.551028 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.551013    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8264777d-c185-4879-8c20-68855787c674-etc-kubernetes\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.555930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.555672    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal" event={"ID":"a79f993c43fd2759c85c24894aebebdd","Type":"ContainerStarted","Data":"34582529689c65da5bbf9e40418eadb6c7d67699de84db10d9ee9731ad4bfbe0"}
Sep 09 16:08:46.557017 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.556913    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z8lm\" (UniqueName: \"kubernetes.io/projected/8264777d-c185-4879-8c20-68855787c674-kube-api-access-8z8lm\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.558203 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.558134    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8264777d-c185-4879-8c20-68855787c674-tmp\") pod \"tuned-lf495\" (UID: \"8264777d-c185-4879-8c20-68855787c674\") " pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.558377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.558353    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv6hx\" (UniqueName: \"kubernetes.io/projected/31eb8e52-4636-4856-a696-385e9a058d49-kube-api-access-bv6hx\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.559722 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.559660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-117.ec2.internal" event={"ID":"12a5a478ae082f068e29cf7483267f02","Type":"ContainerStarted","Data":"4144e94e0198c8fbcbc9de6f38ba46ed65bbd879ff6ebb2c7b4eb68b1c3e4b8c"}
Sep 09 16:08:46.559863 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.559795    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6g9k\" (UniqueName: \"kubernetes.io/projected/0e49d49c-6384-4195-8385-374e3552fc45-kube-api-access-q6g9k\") pod \"node-resolver-5fhrb\" (UID: \"0e49d49c-6384-4195-8385-374e3552fc45\") " pod="openshift-dns/node-resolver-5fhrb"
Sep 09 16:08:46.561728 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.561666    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31eb8e52-4636-4856-a696-385e9a058d49-ovn-node-metrics-cert\") pod \"ovnkube-node-td8ml\" (UID: \"31eb8e52-4636-4856-a696-385e9a058d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.564563 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.564538    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs5mw\" (UniqueName: \"kubernetes.io/projected/96e4e031-e5c2-46a3-a044-0b0d6da36fb1-kube-api-access-cs5mw\") pod \"node-ca-tlk4z\" (UID: \"96e4e031-e5c2-46a3-a044-0b0d6da36fb1\") " pod="openshift-image-registry/node-ca-tlk4z"
Sep 09 16:08:46.564845 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.564821    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvx6w\" (UniqueName: \"kubernetes.io/projected/279dc324-8320-4f43-886a-f152f85a5e4a-kube-api-access-hvx6w\") pod \"iptables-alerter-tm757\" (UID: \"279dc324-8320-4f43-886a-f152f85a5e4a\") " pod="openshift-network-operator/iptables-alerter-tm757"
Sep 09 16:08:46.585632 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.585611    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-cnibin\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7466397e-16a1-4782-89b9-3f2c611e2bd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-registration-dir\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650290    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-device-dir\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-run-k8s-cni-cncf-io\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650353    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/896fdb8e-35d7-4032-8b48-54f32d74e4eb-cni-binary-copy\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650376    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-multus-socket-dir-parent\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650380    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-cnibin\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv99w\" (UniqueName: \"kubernetes.io/projected/7466397e-16a1-4782-89b9-3f2c611e2bd0-kube-api-access-jv99w\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a-konnectivity-ca\") pod \"konnectivity-agent-nrwxd\" (UID: \"dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a\") " pod="kube-system/konnectivity-agent-nrwxd"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650446    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-device-dir\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650453    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-sys-fs\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650479    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5qp\" (UniqueName: \"kubernetes.io/projected/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-kube-api-access-vz5qp\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650508    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/896fdb8e-35d7-4032-8b48-54f32d74e4eb-multus-daemon-config\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650575    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-var-lib-cni-bin\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.653458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-var-lib-kubelet\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7466397e-16a1-4782-89b9-3f2c611e2bd0-cnibin\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-system-cni-dir\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7466397e-16a1-4782-89b9-3f2c611e2bd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-socket-dir\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650809    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-run-k8s-cni-cncf-io\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-run-multus-certs\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651128    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-var-lib-cni-multus\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651186    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-multus-conf-dir\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651237    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzjmz\" (UniqueName: \"kubernetes.io/projected/03652816-9d0a-45d3-9390-04cc6acc8cee-kube-api-access-mzjmz\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651264    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7466397e-16a1-4782-89b9-3f2c611e2bd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a-agent-certs\") pod \"konnectivity-agent-nrwxd\" (UID: \"dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a\") " pod="kube-system/konnectivity-agent-nrwxd"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvlzp\" (UniqueName: \"kubernetes.io/projected/896fdb8e-35d7-4032-8b48-54f32d74e4eb-kube-api-access-wvlzp\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651360    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgp6\" (UniqueName: \"kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6\") pod \"network-check-target-8njz2\" (UID: \"af80a467-9aa6-4c3b-81fd-d05dc3f09e75\") " pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-hostroot\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.654328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7466397e-16a1-4782-89b9-3f2c611e2bd0-system-cni-dir\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651460    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/896fdb8e-35d7-4032-8b48-54f32d74e4eb-cni-binary-copy\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-multus-cni-dir\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-run-netns\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.650508    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-registration-dir\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651647    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-multus-socket-dir-parent\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651647    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-multus-cni-dir\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-etc-kubernetes\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7466397e-16a1-4782-89b9-3f2c611e2bd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651739    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-socket-dir\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651922    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7466397e-16a1-4782-89b9-3f2c611e2bd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.651973    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-sys-fs\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.652211    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-var-lib-cni-multus\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.652256    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-run-multus-certs\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.652301    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.652311    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7466397e-16a1-4782-89b9-3f2c611e2bd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.652419    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.655377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.652451    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a-konnectivity-ca\") pod \"konnectivity-agent-nrwxd\" (UID: \"dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a\") " pod="kube-system/konnectivity-agent-nrwxd"
Sep 09 16:08:46.656040 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.652461    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-multus-conf-dir\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.656040 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.652496    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-var-lib-cni-bin\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.656040 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.652512    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-var-lib-kubelet\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.656040 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.652677    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-hostroot\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.656040 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.652711    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7466397e-16a1-4782-89b9-3f2c611e2bd0-cnibin\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.656040 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:46.652786    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:08:46.656040 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:46.652881    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs podName:03652816-9d0a-45d3-9390-04cc6acc8cee nodeName:}" failed. No retries permitted until 2026-09-09 16:08:47.152861427 +0000 UTC m=+3.088479822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs") pod "network-metrics-daemon-nwfs2" (UID: "03652816-9d0a-45d3-9390-04cc6acc8cee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:08:46.656040 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.652975    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/896fdb8e-35d7-4032-8b48-54f32d74e4eb-multus-daemon-config\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.656040 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.653203    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7466397e-16a1-4782-89b9-3f2c611e2bd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.656040 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.653264    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-system-cni-dir\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.656040 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.653298    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-host-run-netns\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.656040 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.653353    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7466397e-16a1-4782-89b9-3f2c611e2bd0-system-cni-dir\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.656040 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.653454    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896fdb8e-35d7-4032-8b48-54f32d74e4eb-etc-kubernetes\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.656040 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.653612    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7466397e-16a1-4782-89b9-3f2c611e2bd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.657052 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.657027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a-agent-certs\") pod \"konnectivity-agent-nrwxd\" (UID: \"dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a\") " pod="kube-system/konnectivity-agent-nrwxd"
Sep 09 16:08:46.662273 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.662210    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv99w\" (UniqueName: \"kubernetes.io/projected/7466397e-16a1-4782-89b9-3f2c611e2bd0-kube-api-access-jv99w\") pod \"multus-additional-cni-plugins-ghgq7\" (UID: \"7466397e-16a1-4782-89b9-3f2c611e2bd0\") " pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.662441 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:46.662425    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:08:46.662493 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:46.662446    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:08:46.662493 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:46.662460    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zcgp6 for pod openshift-network-diagnostics/network-check-target-8njz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:08:46.662580 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:46.662517    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6 podName:af80a467-9aa6-4c3b-81fd-d05dc3f09e75 nodeName:}" failed. No retries permitted until 2026-09-09 16:08:47.162500235 +0000 UTC m=+3.098118644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zcgp6" (UniqueName: "kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6") pod "network-check-target-8njz2" (UID: "af80a467-9aa6-4c3b-81fd-d05dc3f09e75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:08:46.665038 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.665016    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzjmz\" (UniqueName: \"kubernetes.io/projected/03652816-9d0a-45d3-9390-04cc6acc8cee-kube-api-access-mzjmz\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:46.665824 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.665799    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5qp\" (UniqueName: \"kubernetes.io/projected/681dd1f2-6919-4d7d-8ee1-94f3e46a57f0-kube-api-access-vz5qp\") pod \"aws-ebs-csi-driver-node-kjr6q\" (UID: \"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.666334 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.666286    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvlzp\" (UniqueName: \"kubernetes.io/projected/896fdb8e-35d7-4032-8b48-54f32d74e4eb-kube-api-access-wvlzp\") pod \"multus-b9zp2\" (UID: \"896fdb8e-35d7-4032-8b48-54f32d74e4eb\") " pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.735553 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.735526    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tlk4z"
Sep 09 16:08:46.747167 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:46.746989    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e4e031_e5c2_46a3_a044_0b0d6da36fb1.slice/crio-bf8701fe4367f62e11c88e8e49c69557dcff295bc062b29c1273af8c4abe1f14 WatchSource:0}: Error finding container bf8701fe4367f62e11c88e8e49c69557dcff295bc062b29c1273af8c4abe1f14: Status 404 returned error can't find the container with id bf8701fe4367f62e11c88e8e49c69557dcff295bc062b29c1273af8c4abe1f14
Sep 09 16:08:46.747167 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.747049    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tm757"
Sep 09 16:08:46.757135 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.757114    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:08:46.758364 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:46.758341    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279dc324_8320_4f43_886a_f152f85a5e4a.slice/crio-5f9d235ee782f09f387096d7977d543f1f7dc85ea4d9a7d32ace86b38b0d9d25 WatchSource:0}: Error finding container 5f9d235ee782f09f387096d7977d543f1f7dc85ea4d9a7d32ace86b38b0d9d25: Status 404 returned error can't find the container with id 5f9d235ee782f09f387096d7977d543f1f7dc85ea4d9a7d32ace86b38b0d9d25
Sep 09 16:08:46.764542 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.763888    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lf495"
Sep 09 16:08:46.769281 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:46.769255    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31eb8e52_4636_4856_a696_385e9a058d49.slice/crio-98448516f738830279b8e3e091bb8328e8b4f033a6e77ce91b227f673b0fc3d3 WatchSource:0}: Error finding container 98448516f738830279b8e3e091bb8328e8b4f033a6e77ce91b227f673b0fc3d3: Status 404 returned error can't find the container with id 98448516f738830279b8e3e091bb8328e8b4f033a6e77ce91b227f673b0fc3d3
Sep 09 16:08:46.773495 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.773475    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5fhrb"
Sep 09 16:08:46.774893 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:46.774868    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8264777d_c185_4879_8c20_68855787c674.slice/crio-c891b0c9f3a1fdf2a546f9b8af3a9442a0a8dfaed223544852e536357cd92dfb WatchSource:0}: Error finding container c891b0c9f3a1fdf2a546f9b8af3a9442a0a8dfaed223544852e536357cd92dfb: Status 404 returned error can't find the container with id c891b0c9f3a1fdf2a546f9b8af3a9442a0a8dfaed223544852e536357cd92dfb
Sep 09 16:08:46.781454 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.781254    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b9zp2"
Sep 09 16:08:46.790877 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.790765    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q"
Sep 09 16:08:46.799103 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.799080    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ghgq7"
Sep 09 16:08:46.800437 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:46.800412    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681dd1f2_6919_4d7d_8ee1_94f3e46a57f0.slice/crio-3d1eb5dd5712a654cd9b3ec90dcca43bb4da62f3b81c6dc46b508cd1248e2e83 WatchSource:0}: Error finding container 3d1eb5dd5712a654cd9b3ec90dcca43bb4da62f3b81c6dc46b508cd1248e2e83: Status 404 returned error can't find the container with id 3d1eb5dd5712a654cd9b3ec90dcca43bb4da62f3b81c6dc46b508cd1248e2e83
Sep 09 16:08:46.806606 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.806589    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nrwxd"
Sep 09 16:08:46.962483 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:46.962396    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:08:47.055077 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:47.055031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:47.055257 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:47.055178    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:08:47.055257 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:47.055247    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret podName:27913e87-038f-437f-a49e-6e3768968886 nodeName:}" failed. No retries permitted until 2026-09-09 16:08:48.055228379 +0000 UTC m=+3.990846774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret") pod "global-pull-secret-syncer-psbxz" (UID: "27913e87-038f-437f-a49e-6e3768968886") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:08:47.156055 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:47.156015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:47.156201 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:47.156123    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:08:47.156201 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:47.156200    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs podName:03652816-9d0a-45d3-9390-04cc6acc8cee nodeName:}" failed. No retries permitted until 2026-09-09 16:08:48.156177124 +0000 UTC m=+4.091795524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs") pod "network-metrics-daemon-nwfs2" (UID: "03652816-9d0a-45d3-9390-04cc6acc8cee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:08:47.257402 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:47.257317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgp6\" (UniqueName: \"kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6\") pod \"network-check-target-8njz2\" (UID: \"af80a467-9aa6-4c3b-81fd-d05dc3f09e75\") " pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:08:47.257568 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:47.257471    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:08:47.257568 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:47.257490    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:08:47.257568 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:47.257500    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zcgp6 for pod openshift-network-diagnostics/network-check-target-8njz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:08:47.257568 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:47.257557    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6 podName:af80a467-9aa6-4c3b-81fd-d05dc3f09e75 nodeName:}" failed. No retries permitted until 2026-09-09 16:08:48.2575418 +0000 UTC m=+4.193160217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcgp6" (UniqueName: "kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6") pod "network-check-target-8njz2" (UID: "af80a467-9aa6-4c3b-81fd-d05dc3f09e75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:08:47.473907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:47.473863    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 16:03:46 +0000 UTC" deadline="2028-02-21 23:26:12.436116729 +0000 UTC"
Sep 09 16:08:47.473907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:47.473899    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12727h17m24.96222241s"
Sep 09 16:08:47.547510 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:47.547466    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:08:47.547661 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:47.547601    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:08:47.562375 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:47.562345    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q" event={"ID":"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0","Type":"ContainerStarted","Data":"3d1eb5dd5712a654cd9b3ec90dcca43bb4da62f3b81c6dc46b508cd1248e2e83"}
Sep 09 16:08:47.563622 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:47.563592    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lf495" event={"ID":"8264777d-c185-4879-8c20-68855787c674","Type":"ContainerStarted","Data":"c891b0c9f3a1fdf2a546f9b8af3a9442a0a8dfaed223544852e536357cd92dfb"}
Sep 09 16:08:47.564918 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:47.564896    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tlk4z" event={"ID":"96e4e031-e5c2-46a3-a044-0b0d6da36fb1","Type":"ContainerStarted","Data":"bf8701fe4367f62e11c88e8e49c69557dcff295bc062b29c1273af8c4abe1f14"}
Sep 09 16:08:47.566243 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:47.566209    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b9zp2" event={"ID":"896fdb8e-35d7-4032-8b48-54f32d74e4eb","Type":"ContainerStarted","Data":"fa17a9c76ef667debeef5806e9922f204351f2e5843d237f7fea7136a42d3cc6"}
Sep 09 16:08:47.567525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:47.567502    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5fhrb" event={"ID":"0e49d49c-6384-4195-8385-374e3552fc45","Type":"ContainerStarted","Data":"6090e2b265751be9e3601221705107e240c9d74b848350386498210f4d0d934a"}
Sep 09 16:08:47.569007 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:47.568974    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml" event={"ID":"31eb8e52-4636-4856-a696-385e9a058d49","Type":"ContainerStarted","Data":"98448516f738830279b8e3e091bb8328e8b4f033a6e77ce91b227f673b0fc3d3"}
Sep 09 16:08:47.570416 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:47.570391    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tm757" event={"ID":"279dc324-8320-4f43-886a-f152f85a5e4a","Type":"ContainerStarted","Data":"5f9d235ee782f09f387096d7977d543f1f7dc85ea4d9a7d32ace86b38b0d9d25"}
Sep 09 16:08:47.993110 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:08:47.993013    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7466397e_16a1_4782_89b9_3f2c611e2bd0.slice/crio-4751e1b1eedd50f9621f0304da6f579a7dd4d9c88ccff47821e7f1e24078a8b0 WatchSource:0}: Error finding container 4751e1b1eedd50f9621f0304da6f579a7dd4d9c88ccff47821e7f1e24078a8b0: Status 404 returned error can't find the container with id 4751e1b1eedd50f9621f0304da6f579a7dd4d9c88ccff47821e7f1e24078a8b0
Sep 09 16:08:48.064704 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:48.064603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:48.064852 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:48.064743    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:08:48.064852 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:48.064808    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret podName:27913e87-038f-437f-a49e-6e3768968886 nodeName:}" failed. No retries permitted until 2026-09-09 16:08:50.064787642 +0000 UTC m=+6.000406039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret") pod "global-pull-secret-syncer-psbxz" (UID: "27913e87-038f-437f-a49e-6e3768968886") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:08:48.165378 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:48.165339    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:48.165546 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:48.165511    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:08:48.165616 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:48.165576    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs podName:03652816-9d0a-45d3-9390-04cc6acc8cee nodeName:}" failed. No retries permitted until 2026-09-09 16:08:50.165557269 +0000 UTC m=+6.101175664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs") pod "network-metrics-daemon-nwfs2" (UID: "03652816-9d0a-45d3-9390-04cc6acc8cee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:08:48.266339 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:48.266249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgp6\" (UniqueName: \"kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6\") pod \"network-check-target-8njz2\" (UID: \"af80a467-9aa6-4c3b-81fd-d05dc3f09e75\") " pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:08:48.266493 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:48.266426    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:08:48.266493 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:48.266458    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:08:48.266493 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:48.266471    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zcgp6 for pod openshift-network-diagnostics/network-check-target-8njz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:08:48.266664 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:48.266547    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6 podName:af80a467-9aa6-4c3b-81fd-d05dc3f09e75 nodeName:}" failed. No retries permitted until 2026-09-09 16:08:50.266526044 +0000 UTC m=+6.202144460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcgp6" (UniqueName: "kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6") pod "network-check-target-8njz2" (UID: "af80a467-9aa6-4c3b-81fd-d05dc3f09e75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:08:48.474169 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:48.474060    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 16:03:46 +0000 UTC" deadline="2028-06-10 00:30:35.19853725 +0000 UTC"
Sep 09 16:08:48.474169 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:48.474127    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15344h21m46.724414967s"
Sep 09 16:08:48.549198 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:48.549076    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:48.549280 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:48.549205    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:08:48.549342 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:48.549304    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:48.549574 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:48.549434    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:08:48.576358 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:48.576321    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal" event={"ID":"a79f993c43fd2759c85c24894aebebdd","Type":"ContainerStarted","Data":"8a408954bc1f90c7556417ab8bbac5efe9b0ae6fe6ac51bad004e34fa34522df"}
Sep 09 16:08:48.578389 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:48.578325    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nrwxd" event={"ID":"dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a","Type":"ContainerStarted","Data":"0dbf313c21074442d05d577d37473d8368435776954c974649b15f0f059b2c8c"}
Sep 09 16:08:48.580662 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:48.580634    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghgq7" event={"ID":"7466397e-16a1-4782-89b9-3f2c611e2bd0","Type":"ContainerStarted","Data":"4751e1b1eedd50f9621f0304da6f579a7dd4d9c88ccff47821e7f1e24078a8b0"}
Sep 09 16:08:48.583258 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:48.582811    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-117.ec2.internal" event={"ID":"12a5a478ae082f068e29cf7483267f02","Type":"ContainerStarted","Data":"9f926aaf02989c3632a4dbb3461284fc5c31393bcfe77debdfc1ced488df6b65"}
Sep 09 16:08:49.547813 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:49.547762    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:08:49.548235 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:49.547859    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:08:49.587934 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:49.587391    2576 generic.go:356] "Generic (PLEG): container finished" podID="a79f993c43fd2759c85c24894aebebdd" containerID="8a408954bc1f90c7556417ab8bbac5efe9b0ae6fe6ac51bad004e34fa34522df" exitCode=0
Sep 09 16:08:49.587934 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:49.587876    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal" event={"ID":"a79f993c43fd2759c85c24894aebebdd","Type":"ContainerDied","Data":"8a408954bc1f90c7556417ab8bbac5efe9b0ae6fe6ac51bad004e34fa34522df"}
Sep 09 16:08:49.601658 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:49.601599    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-117.ec2.internal" podStartSLOduration=3.601580479 podStartE2EDuration="3.601580479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:08:48.6068086 +0000 UTC m=+4.542427040" watchObservedRunningTime="2026-09-09 16:08:49.601580479 +0000 UTC m=+5.537198894"
Sep 09 16:08:50.080525 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:50.079963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:50.080525 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:50.080134    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:08:50.080525 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:50.080192    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret podName:27913e87-038f-437f-a49e-6e3768968886 nodeName:}" failed. No retries permitted until 2026-09-09 16:08:54.080174429 +0000 UTC m=+10.015792838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret") pod "global-pull-secret-syncer-psbxz" (UID: "27913e87-038f-437f-a49e-6e3768968886") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:08:50.180898 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:50.180861    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:50.181069 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:50.180985    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:08:50.181069 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:50.181054    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs podName:03652816-9d0a-45d3-9390-04cc6acc8cee nodeName:}" failed. No retries permitted until 2026-09-09 16:08:54.18103908 +0000 UTC m=+10.116657479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs") pod "network-metrics-daemon-nwfs2" (UID: "03652816-9d0a-45d3-9390-04cc6acc8cee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:08:50.281994 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:50.281955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgp6\" (UniqueName: \"kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6\") pod \"network-check-target-8njz2\" (UID: \"af80a467-9aa6-4c3b-81fd-d05dc3f09e75\") " pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:08:50.282164 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:50.282145    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:08:50.282214 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:50.282170    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:08:50.282214 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:50.282182    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zcgp6 for pod openshift-network-diagnostics/network-check-target-8njz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:08:50.282277 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:50.282237    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6 podName:af80a467-9aa6-4c3b-81fd-d05dc3f09e75 nodeName:}" failed. No retries permitted until 2026-09-09 16:08:54.282224003 +0000 UTC m=+10.217842412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcgp6" (UniqueName: "kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6") pod "network-check-target-8njz2" (UID: "af80a467-9aa6-4c3b-81fd-d05dc3f09e75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:08:50.554038 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:50.553963    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:50.554417 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:50.554084    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:08:50.554473 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:50.554443    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:50.554554 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:50.554533    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:08:51.548143 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:51.548104    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:08:51.548347 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:51.548234    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:08:52.551201 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:52.548129    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:52.551201 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:52.548271    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:08:52.551201 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:52.548648    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:52.551201 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:52.548768    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:08:53.548098 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:53.547820    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:08:53.548260 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:53.548151    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:08:53.599619 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:53.599584    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5fhrb" event={"ID":"0e49d49c-6384-4195-8385-374e3552fc45","Type":"ContainerStarted","Data":"57fd7eb52d40dd67bd5a16bdd2e27a00b80e321c34cac64b7519cc824c4d4f88"}
Sep 09 16:08:53.603142 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:53.603115    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal" event={"ID":"a79f993c43fd2759c85c24894aebebdd","Type":"ContainerStarted","Data":"42af18827c47ce4cfbbb8775bfeb52f25fc8276a195afd548dc72a6b96f353ab"}
Sep 09 16:08:53.606034 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:53.606011    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q" event={"ID":"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0","Type":"ContainerStarted","Data":"3ba8c31207395857e26bbe7c72870145f200c0908acedabf4f5b9823e60eba09"}
Sep 09 16:08:53.609397 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:53.609355    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tlk4z" event={"ID":"96e4e031-e5c2-46a3-a044-0b0d6da36fb1","Type":"ContainerStarted","Data":"0ca7297be5d7b32a9c3bc4352a9ac0a900d1305484e9d464abc7bf82c605ed1b"}
Sep 09 16:08:53.615259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:53.614806    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghgq7" event={"ID":"7466397e-16a1-4782-89b9-3f2c611e2bd0","Type":"ContainerStarted","Data":"b90ed26285b4fe90eebf68041de196d8d2e4872f1ac319ebb034f9a1037b1994"}
Sep 09 16:08:53.631390 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:53.630496    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5fhrb" podStartSLOduration=3.416198489 podStartE2EDuration="9.630484165s" totalImagesPullingTime="6.214285676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:08:53.613990617 +0000 UTC m=+9.549609063" watchObservedRunningTime="2026-09-09 16:08:53.630484165 +0000 UTC m=+9.566102581"
Sep 09 16:08:53.631390 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:53.630561    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tlk4z" podStartSLOduration=3.4069123709999998 podStartE2EDuration="9.630556985s" totalImagesPullingTime="6.223644614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:08:53.630247232 +0000 UTC m=+9.565865668" watchObservedRunningTime="2026-09-09 16:08:53.630556985 +0000 UTC m=+9.566175411"
Sep 09 16:08:53.668441 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:53.667647    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-117.ec2.internal" podStartSLOduration=7.667628306 podStartE2EDuration="7.667628306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:08:53.64488362 +0000 UTC m=+9.580502039" watchObservedRunningTime="2026-09-09 16:08:53.667628306 +0000 UTC m=+9.603246723"
Sep 09 16:08:54.116783 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:54.116633    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:54.117470 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:54.117050    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:08:54.117470 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:54.117134    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret podName:27913e87-038f-437f-a49e-6e3768968886 nodeName:}" failed. No retries permitted until 2026-09-09 16:09:02.117115412 +0000 UTC m=+18.052733819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret") pod "global-pull-secret-syncer-psbxz" (UID: "27913e87-038f-437f-a49e-6e3768968886") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:08:54.219254 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:54.218665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:54.219254 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:54.218852    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:08:54.219254 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:54.218917    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs podName:03652816-9d0a-45d3-9390-04cc6acc8cee nodeName:}" failed. No retries permitted until 2026-09-09 16:09:02.218899096 +0000 UTC m=+18.154517492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs") pod "network-metrics-daemon-nwfs2" (UID: "03652816-9d0a-45d3-9390-04cc6acc8cee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:08:54.319581 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:54.319546    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgp6\" (UniqueName: \"kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6\") pod \"network-check-target-8njz2\" (UID: \"af80a467-9aa6-4c3b-81fd-d05dc3f09e75\") " pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:08:54.319763 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:54.319708    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:08:54.319763 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:54.319731    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:08:54.319763 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:54.319744    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zcgp6 for pod openshift-network-diagnostics/network-check-target-8njz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:08:54.319936 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:54.319813    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6 podName:af80a467-9aa6-4c3b-81fd-d05dc3f09e75 nodeName:}" failed. No retries permitted until 2026-09-09 16:09:02.319792409 +0000 UTC m=+18.255410828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcgp6" (UniqueName: "kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6") pod "network-check-target-8njz2" (UID: "af80a467-9aa6-4c3b-81fd-d05dc3f09e75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:08:54.548731 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:54.548685    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:54.548890 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:54.548804    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:08:54.549115 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:54.549096    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:54.549228 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:54.549202    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:08:55.548204 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:55.548162    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:08:55.548653 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:55.548286    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:08:55.627844 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:55.627806    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tm757" event={"ID":"279dc324-8320-4f43-886a-f152f85a5e4a","Type":"ContainerStarted","Data":"1ba0c64427d55c13b3bde6695b6ab04abb7a51c1e1723b7925137147488e4ddf"}
Sep 09 16:08:55.630295 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:55.630264    2576 generic.go:356] "Generic (PLEG): container finished" podID="7466397e-16a1-4782-89b9-3f2c611e2bd0" containerID="b90ed26285b4fe90eebf68041de196d8d2e4872f1ac319ebb034f9a1037b1994" exitCode=0
Sep 09 16:08:55.630417 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:55.630295    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghgq7" event={"ID":"7466397e-16a1-4782-89b9-3f2c611e2bd0","Type":"ContainerDied","Data":"b90ed26285b4fe90eebf68041de196d8d2e4872f1ac319ebb034f9a1037b1994"}
Sep 09 16:08:55.676418 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:55.676362    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tm757" podStartSLOduration=5.450467068 podStartE2EDuration="11.676340771s" totalImagesPullingTime="6.225873703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:08:55.642328294 +0000 UTC m=+11.577946728" watchObservedRunningTime="2026-09-09 16:08:55.676340771 +0000 UTC m=+11.611959187"
Sep 09 16:08:56.547244 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:56.547208    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:56.547407 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:56.547266    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:56.547407 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:56.547397    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:08:56.547530 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:56.547498    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:08:57.547541 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:57.547496    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:08:57.548021 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:57.547625    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:08:58.548040 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:58.548008    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:08:58.548462 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:58.548155    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:08:58.548511 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:58.548484    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:08:58.548588 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:58.548569    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:08:59.548080 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:08:59.548048    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:08:59.548535 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:08:59.548169    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:09:00.547194 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:00.547158    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:00.547374 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:00.547267    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:09:00.547374 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:00.547321    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:09:00.547497 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:00.547373    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:09:01.548057 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:01.548024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:01.548507 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:01.548155    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:09:02.181784 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:02.181748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:02.181915 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:02.181854    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:09:02.181915 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:02.181909    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret podName:27913e87-038f-437f-a49e-6e3768968886 nodeName:}" failed. No retries permitted until 2026-09-09 16:09:18.181895775 +0000 UTC m=+34.117514168 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret") pod "global-pull-secret-syncer-psbxz" (UID: "27913e87-038f-437f-a49e-6e3768968886") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:09:02.282077 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:02.282040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:09:02.282253 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:02.282168    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:09:02.282253 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:02.282232    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs podName:03652816-9d0a-45d3-9390-04cc6acc8cee nodeName:}" failed. No retries permitted until 2026-09-09 16:09:18.282214733 +0000 UTC m=+34.217833140 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs") pod "network-metrics-daemon-nwfs2" (UID: "03652816-9d0a-45d3-9390-04cc6acc8cee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:09:02.383185 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:02.383150    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgp6\" (UniqueName: \"kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6\") pod \"network-check-target-8njz2\" (UID: \"af80a467-9aa6-4c3b-81fd-d05dc3f09e75\") " pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:02.383361 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:02.383311    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:09:02.383361 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:02.383339    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:09:02.383361 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:02.383350    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zcgp6 for pod openshift-network-diagnostics/network-check-target-8njz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:09:02.383503 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:02.383409    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6 podName:af80a467-9aa6-4c3b-81fd-d05dc3f09e75 nodeName:}" failed. No retries permitted until 2026-09-09 16:09:18.383394452 +0000 UTC m=+34.319012849 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcgp6" (UniqueName: "kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6") pod "network-check-target-8njz2" (UID: "af80a467-9aa6-4c3b-81fd-d05dc3f09e75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:09:02.548169 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:02.548085    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:09:02.548555 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:02.548085    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:02.548555 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:02.548234    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:09:02.548555 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:02.548269    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:09:03.547426 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:03.547382    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:03.547610 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:03.547559    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:09:04.547944 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:04.547912    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:04.548349 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:04.547995    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:09:04.548349 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:04.548002    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:09:04.548349 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:04.548112    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:09:05.547403 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:05.547371    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:05.547542 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:05.547488    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:09:06.547253 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:06.547224    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:09:06.547709 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:06.547224    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:06.547709 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:06.547339    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:09:06.547709 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:06.547395    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:09:07.548199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:07.548154    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:07.548620 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:07.548301    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:09:07.943413 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:07.943218    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 16:09:08.506984 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.506619    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T16:09:07.943398715Z","UUID":"8accabe5-8d23-4fb5-a894-4259e9ec05fd","Handler":null,"Name":"","Endpoint":""}
Sep 09 16:09:08.508033 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.508017    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 16:09:08.508033 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.508045    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 16:09:08.548043 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.548014    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:09:08.548043 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.548047    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:08.548235 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:08.548124    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:09:08.548791 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:08.548264    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:09:08.657820 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.657782    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q" event={"ID":"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0","Type":"ContainerStarted","Data":"14f8301be4e4f96c5427ae141abec1ef6f1dfb76761931f104dcfe85efaac7e0"}
Sep 09 16:09:08.659278 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.659251    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lf495" event={"ID":"8264777d-c185-4879-8c20-68855787c674","Type":"ContainerStarted","Data":"6d43204c3e250650228d8b255812862746119703446a4f12b6d9bfc592f8e1a1"}
Sep 09 16:09:08.660883 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.660861    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nrwxd" event={"ID":"dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a","Type":"ContainerStarted","Data":"b2fc3c0308f53ff0c1e06a5a2f40e560c123433b6501057f834040ddf1ed2a95"}
Sep 09 16:09:08.666179 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.666158    2576 generic.go:356] "Generic (PLEG): container finished" podID="7466397e-16a1-4782-89b9-3f2c611e2bd0" containerID="c6a852acff74a7149b58e419b4babcdbf2c66a2fa22551086212015872b4b4a7" exitCode=0
Sep 09 16:09:08.666255 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.666223    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghgq7" event={"ID":"7466397e-16a1-4782-89b9-3f2c611e2bd0","Type":"ContainerDied","Data":"c6a852acff74a7149b58e419b4babcdbf2c66a2fa22551086212015872b4b4a7"}
Sep 09 16:09:08.667871 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.667857    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b9zp2" event={"ID":"896fdb8e-35d7-4032-8b48-54f32d74e4eb","Type":"ContainerStarted","Data":"b077e649eee12c0404a89227bd12c880e702ca4e2ad6ae8fb48eb2b2e39d7083"}
Sep 09 16:09:08.670864 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.670844    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml" event={"ID":"31eb8e52-4636-4856-a696-385e9a058d49","Type":"ContainerStarted","Data":"3e06fb196f0dba6880a07faf5536e7c9357d4aa9cd1af6e20ec65ac4400eb8eb"}
Sep 09 16:09:08.670946 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.670912    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml" event={"ID":"31eb8e52-4636-4856-a696-385e9a058d49","Type":"ContainerStarted","Data":"bb68e41381f93b8d5c38502f634533df8017a61d774bb773d03320cce2142b0d"}
Sep 09 16:09:08.670946 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.670945    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml" event={"ID":"31eb8e52-4636-4856-a696-385e9a058d49","Type":"ContainerStarted","Data":"3ecb7dd6801b292b3e4edba1185ca70eb1e59c176be8fd91edc9d5cfdbce05cc"}
Sep 09 16:09:08.671013 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.670973    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml" event={"ID":"31eb8e52-4636-4856-a696-385e9a058d49","Type":"ContainerStarted","Data":"6dcc1582c9aeb8a59836670466b76e57bddccae6668f6587a5ef1fa4fb224ba8"}
Sep 09 16:09:08.671013 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.671005    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml" event={"ID":"31eb8e52-4636-4856-a696-385e9a058d49","Type":"ContainerStarted","Data":"c7ae96467f66ba4073f4719c6ecbbae6aaeceb61b41519c426275a9fd70a78c7"}
Sep 09 16:09:08.671074 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.671035    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml" event={"ID":"31eb8e52-4636-4856-a696-385e9a058d49","Type":"ContainerStarted","Data":"a1820e695679c71230a51f4d1f4ca6cb704ef8fec734a207d62f95bba4b3216f"}
Sep 09 16:09:08.677208 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.677174    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lf495" podStartSLOduration=3.6677760360000002 podStartE2EDuration="24.677162525s" totalImagesPullingTime="21.009386489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:08.676749375 +0000 UTC m=+24.612367802" watchObservedRunningTime="2026-09-09 16:09:08.677162525 +0000 UTC m=+24.612780950"
Sep 09 16:09:08.692242 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.692211    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-b9zp2" podStartSLOduration=3.698221581 podStartE2EDuration="24.692200561s" totalImagesPullingTime="20.99397898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:08.691584477 +0000 UTC m=+24.627202905" watchObservedRunningTime="2026-09-09 16:09:08.692200561 +0000 UTC m=+24.627818976"
Sep 09 16:09:08.722512 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:08.722483    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nrwxd" podStartSLOduration=4.938960586 podStartE2EDuration="24.72247143s" totalImagesPullingTime="19.783510844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:08.722276844 +0000 UTC m=+24.657895272" watchObservedRunningTime="2026-09-09 16:09:08.72247143 +0000 UTC m=+24.658089845"
Sep 09 16:09:09.548042 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:09.548023    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:09.548127 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:09.548116    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:09:09.674967 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:09.674896    2576 generic.go:356] "Generic (PLEG): container finished" podID="7466397e-16a1-4782-89b9-3f2c611e2bd0" containerID="7893b78da2e8441c949e11b597e39b67fa32da827b0f985a763908e45ab773de" exitCode=0
Sep 09 16:09:09.675289 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:09.675001    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghgq7" event={"ID":"7466397e-16a1-4782-89b9-3f2c611e2bd0","Type":"ContainerDied","Data":"7893b78da2e8441c949e11b597e39b67fa32da827b0f985a763908e45ab773de"}
Sep 09 16:09:10.547431 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:10.547412    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:10.547528 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:10.547509    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:09:10.547582 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:10.547564    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:09:10.547672 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:10.547657    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:09:10.682050 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:10.681975    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml" event={"ID":"31eb8e52-4636-4856-a696-385e9a058d49","Type":"ContainerStarted","Data":"2e2c9f6ac930935018c7de4ae6e1f2422dc21fa3607ec565cb7981938076a9c9"}
Sep 09 16:09:11.547413 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:11.547337    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:11.547542 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:11.547429    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:09:11.678978 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:11.678949    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nrwxd"
Sep 09 16:09:11.679537 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:11.679522    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nrwxd"
Sep 09 16:09:11.686547 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:11.686522    2576 generic.go:356] "Generic (PLEG): container finished" podID="7466397e-16a1-4782-89b9-3f2c611e2bd0" containerID="bb2a15313c769265d80fc14abfa4aacfba47ba2dcf6be6a25a3f9c2fcfb097b7" exitCode=0
Sep 09 16:09:11.686960 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:11.686595    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghgq7" event={"ID":"7466397e-16a1-4782-89b9-3f2c611e2bd0","Type":"ContainerDied","Data":"bb2a15313c769265d80fc14abfa4aacfba47ba2dcf6be6a25a3f9c2fcfb097b7"}
Sep 09 16:09:11.688814 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:11.688792    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q" event={"ID":"681dd1f2-6919-4d7d-8ee1-94f3e46a57f0","Type":"ContainerStarted","Data":"c9f73a952f2493fe092628a88e306a7fb973c1706d76bad6601fd2936da09fd0"}
Sep 09 16:09:11.689072 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:11.689058    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nrwxd"
Sep 09 16:09:11.689333 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:11.689316    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nrwxd"
Sep 09 16:09:11.774413 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:11.774373    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjr6q" podStartSLOduration=3.705416029 podStartE2EDuration="27.774358975s" totalImagesPullingTime="24.068942946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:08:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:11.739476523 +0000 UTC m=+27.675094951" watchObservedRunningTime="2026-09-09 16:09:11.774358975 +0000 UTC m=+27.709977392"
Sep 09 16:09:12.547444 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:12.547416    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:09:12.547609 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:12.547515    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:09:12.547609 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:12.547569    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:12.547712 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:12.547641    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:09:12.700672 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:12.700521    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml" event={"ID":"31eb8e52-4636-4856-a696-385e9a058d49","Type":"ContainerStarted","Data":"69b5237f3cccf528f790e758cda631a1ca4f4ff418f839d9812916e3f2d04bba"}
Sep 09 16:09:12.731815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:12.731775    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml" podStartSLOduration=7.716818044 podStartE2EDuration="28.73176006s" totalImagesPullingTime="21.014942016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:12.731605146 +0000 UTC m=+28.667223562" watchObservedRunningTime="2026-09-09 16:09:12.73176006 +0000 UTC m=+28.667378476"
Sep 09 16:09:13.547712 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:13.547551    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:13.547856 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:13.547780    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:09:13.702451 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:13.702418    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:09:13.702834 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:13.702504    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:09:13.702834 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:13.702556    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:09:13.717887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:13.717860    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:09:13.718009 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:13.717958    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:09:14.521808 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:14.521781    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8njz2"]
Sep 09 16:09:14.521984 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:14.521869    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:14.521984 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:14.521946    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:09:14.525320 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:14.525238    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-psbxz"]
Sep 09 16:09:14.525459 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:14.525344    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:14.525459 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:14.525443    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:09:14.525947 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:14.525925    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nwfs2"]
Sep 09 16:09:14.526052 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:14.526006    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:09:14.526109 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:14.526086    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:09:16.547554 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:16.547517    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:16.548319 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:16.547560    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:09:16.548319 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:16.547520    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:16.548319 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:16.547648    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:09:16.548319 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:16.547837    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:09:16.548319 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:16.547928    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:09:18.201665 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:18.201635    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:18.202094 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:18.201794    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:09:18.202094 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:18.201864    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret podName:27913e87-038f-437f-a49e-6e3768968886 nodeName:}" failed. No retries permitted until 2026-09-09 16:09:50.201847493 +0000 UTC m=+66.137465887 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret") pod "global-pull-secret-syncer-psbxz" (UID: "27913e87-038f-437f-a49e-6e3768968886") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:09:18.302841 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:18.302713    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:09:18.302954 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:18.302796    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:09:18.302954 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:18.302914    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs podName:03652816-9d0a-45d3-9390-04cc6acc8cee nodeName:}" failed. No retries permitted until 2026-09-09 16:09:50.302893884 +0000 UTC m=+66.238512292 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs") pod "network-metrics-daemon-nwfs2" (UID: "03652816-9d0a-45d3-9390-04cc6acc8cee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:09:18.403244 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:18.403214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgp6\" (UniqueName: \"kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6\") pod \"network-check-target-8njz2\" (UID: \"af80a467-9aa6-4c3b-81fd-d05dc3f09e75\") " pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:18.403400 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:18.403381    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:09:18.403437 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:18.403406    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:09:18.403437 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:18.403416    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zcgp6 for pod openshift-network-diagnostics/network-check-target-8njz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:09:18.403521 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:18.403462    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6 podName:af80a467-9aa6-4c3b-81fd-d05dc3f09e75 nodeName:}" failed. No retries permitted until 2026-09-09 16:09:50.403448985 +0000 UTC m=+66.339067383 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcgp6" (UniqueName: "kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6") pod "network-check-target-8njz2" (UID: "af80a467-9aa6-4c3b-81fd-d05dc3f09e75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:09:18.547324 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:18.547286    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:18.547324 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:18.547318    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:09:18.547502 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:18.547292    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:18.547502 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:18.547405    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8njz2" podUID="af80a467-9aa6-4c3b-81fd-d05dc3f09e75"
Sep 09 16:09:18.547787 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:18.547501    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:09:18.547787 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:18.547594    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbxz" podUID="27913e87-038f-437f-a49e-6e3768968886"
Sep 09 16:09:18.717226 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:18.717151    2576 generic.go:356] "Generic (PLEG): container finished" podID="7466397e-16a1-4782-89b9-3f2c611e2bd0" containerID="146864aa753306060dda3f5b6c8d84b18f306a5d467fd5d8f1207dbac762733b" exitCode=0
Sep 09 16:09:18.717226 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:18.717217    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghgq7" event={"ID":"7466397e-16a1-4782-89b9-3f2c611e2bd0","Type":"ContainerDied","Data":"146864aa753306060dda3f5b6c8d84b18f306a5d467fd5d8f1207dbac762733b"}
Sep 09 16:09:19.722288 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:19.722256    2576 generic.go:356] "Generic (PLEG): container finished" podID="7466397e-16a1-4782-89b9-3f2c611e2bd0" containerID="eadd84770feb6a8dd53a51a4d02ab4ac123429dd020c91cbbb0b8304f656481e" exitCode=0
Sep 09 16:09:19.722648 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:19.722307    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghgq7" event={"ID":"7466397e-16a1-4782-89b9-3f2c611e2bd0","Type":"ContainerDied","Data":"eadd84770feb6a8dd53a51a4d02ab4ac123429dd020c91cbbb0b8304f656481e"}
Sep 09 16:09:20.351618 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.351597    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-117.ec2.internal" event="NodeReady"
Sep 09 16:09:20.351816 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.351724    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 16:09:20.402087 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.402059    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9s8wl"]
Sep 09 16:09:20.415998 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.415960    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bmprr"]
Sep 09 16:09:20.416128 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.416109    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:20.421273 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.421251    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 16:09:20.421432 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.421383    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 16:09:20.421497 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.421382    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pgn4p\""
Sep 09 16:09:20.426226 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.425903    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9s8wl"]
Sep 09 16:09:20.426226 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.425979    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bmprr"]
Sep 09 16:09:20.426226 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.426075    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bmprr"
Sep 09 16:09:20.429175 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.429157    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-74c5r\""
Sep 09 16:09:20.430509 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.430491    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 16:09:20.430573 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.430554    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 16:09:20.430610 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.430570    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 16:09:20.522016 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.521949    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7hlm\" (UniqueName: \"kubernetes.io/projected/f38b903a-197a-466f-87d5-e3fac326978a-kube-api-access-v7hlm\") pod \"ingress-canary-bmprr\" (UID: \"f38b903a-197a-466f-87d5-e3fac326978a\") " pod="openshift-ingress-canary/ingress-canary-bmprr"
Sep 09 16:09:20.522016 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.522003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f38b903a-197a-466f-87d5-e3fac326978a-cert\") pod \"ingress-canary-bmprr\" (UID: \"f38b903a-197a-466f-87d5-e3fac326978a\") " pod="openshift-ingress-canary/ingress-canary-bmprr"
Sep 09 16:09:20.522163 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.522079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:20.522163 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.522102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19683a66-1e9f-4c96-85bf-5291daa9d7b5-tmp-dir\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:20.522163 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.522142    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xx2k\" (UniqueName: \"kubernetes.io/projected/19683a66-1e9f-4c96-85bf-5291daa9d7b5-kube-api-access-7xx2k\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:20.522294 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.522165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19683a66-1e9f-4c96-85bf-5291daa9d7b5-config-volume\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:20.547118 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.547101    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:20.547202 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.547101    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:20.547264 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.547108    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:09:20.554209 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.554192    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 16:09:20.555029 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.555014    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zj7bv\""
Sep 09 16:09:20.555367 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.555351    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 16:09:20.555449 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.555383    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-trsmq\""
Sep 09 16:09:20.559612 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.559592    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 16:09:20.560285 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.560267    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 16:09:20.623063 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.623039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:20.623063 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.623060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19683a66-1e9f-4c96-85bf-5291daa9d7b5-tmp-dir\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:20.623190 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:20.623145    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:09:20.623226 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.623186    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xx2k\" (UniqueName: \"kubernetes.io/projected/19683a66-1e9f-4c96-85bf-5291daa9d7b5-kube-api-access-7xx2k\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:20.623226 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:20.623200    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls podName:19683a66-1e9f-4c96-85bf-5291daa9d7b5 nodeName:}" failed. No retries permitted until 2026-09-09 16:09:21.123185724 +0000 UTC m=+37.058804118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls") pod "dns-default-9s8wl" (UID: "19683a66-1e9f-4c96-85bf-5291daa9d7b5") : secret "dns-default-metrics-tls" not found
Sep 09 16:09:20.623326 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.623231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19683a66-1e9f-4c96-85bf-5291daa9d7b5-config-volume\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:20.623326 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.623259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7hlm\" (UniqueName: \"kubernetes.io/projected/f38b903a-197a-466f-87d5-e3fac326978a-kube-api-access-v7hlm\") pod \"ingress-canary-bmprr\" (UID: \"f38b903a-197a-466f-87d5-e3fac326978a\") " pod="openshift-ingress-canary/ingress-canary-bmprr"
Sep 09 16:09:20.623429 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.623340    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19683a66-1e9f-4c96-85bf-5291daa9d7b5-tmp-dir\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:20.623429 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.623389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f38b903a-197a-466f-87d5-e3fac326978a-cert\") pod \"ingress-canary-bmprr\" (UID: \"f38b903a-197a-466f-87d5-e3fac326978a\") " pod="openshift-ingress-canary/ingress-canary-bmprr"
Sep 09 16:09:20.623806 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.623789    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19683a66-1e9f-4c96-85bf-5291daa9d7b5-config-volume\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:20.627357 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.627335    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f38b903a-197a-466f-87d5-e3fac326978a-cert\") pod \"ingress-canary-bmprr\" (UID: \"f38b903a-197a-466f-87d5-e3fac326978a\") " pod="openshift-ingress-canary/ingress-canary-bmprr"
Sep 09 16:09:20.632006 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.631984    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xx2k\" (UniqueName: \"kubernetes.io/projected/19683a66-1e9f-4c96-85bf-5291daa9d7b5-kube-api-access-7xx2k\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:20.632085 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.632031    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7hlm\" (UniqueName: \"kubernetes.io/projected/f38b903a-197a-466f-87d5-e3fac326978a-kube-api-access-v7hlm\") pod \"ingress-canary-bmprr\" (UID: \"f38b903a-197a-466f-87d5-e3fac326978a\") " pod="openshift-ingress-canary/ingress-canary-bmprr"
Sep 09 16:09:20.728206 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.728175    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghgq7" event={"ID":"7466397e-16a1-4782-89b9-3f2c611e2bd0","Type":"ContainerStarted","Data":"180fc7137faae743d8326e230410fda70e6408f7c5470ad838c78a0f8085a8d4"}
Sep 09 16:09:20.735811 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.735790    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bmprr"
Sep 09 16:09:20.750988 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:20.750951    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ghgq7" podStartSLOduration=11.248739806 podStartE2EDuration="36.750938103s" totalImagesPullingTime="25.502198297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:08:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:20.750055046 +0000 UTC m=+36.685673473" watchObservedRunningTime="2026-09-09 16:09:20.750938103 +0000 UTC m=+36.686556500"
Sep 09 16:09:21.063868 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:21.063791    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bmprr"]
Sep 09 16:09:21.067747 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:09:21.067720    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38b903a_197a_466f_87d5_e3fac326978a.slice/crio-9244b947b93e175e47ed1b404d95b4493b5d974cce17d00f08f7702144256821 WatchSource:0}: Error finding container 9244b947b93e175e47ed1b404d95b4493b5d974cce17d00f08f7702144256821: Status 404 returned error can't find the container with id 9244b947b93e175e47ed1b404d95b4493b5d974cce17d00f08f7702144256821
Sep 09 16:09:21.126652 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:21.126461    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:21.126798 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:21.126606    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:09:21.126798 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:21.126777    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls podName:19683a66-1e9f-4c96-85bf-5291daa9d7b5 nodeName:}" failed. No retries permitted until 2026-09-09 16:09:22.126758066 +0000 UTC m=+38.062376461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls") pod "dns-default-9s8wl" (UID: "19683a66-1e9f-4c96-85bf-5291daa9d7b5") : secret "dns-default-metrics-tls" not found
Sep 09 16:09:21.731375 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:21.731340    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bmprr" event={"ID":"f38b903a-197a-466f-87d5-e3fac326978a","Type":"ContainerStarted","Data":"9244b947b93e175e47ed1b404d95b4493b5d974cce17d00f08f7702144256821"}
Sep 09 16:09:22.134244 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:22.134164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:22.134426 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:22.134342    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:09:22.134471 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:22.134427    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls podName:19683a66-1e9f-4c96-85bf-5291daa9d7b5 nodeName:}" failed. No retries permitted until 2026-09-09 16:09:24.134404302 +0000 UTC m=+40.070022717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls") pod "dns-default-9s8wl" (UID: "19683a66-1e9f-4c96-85bf-5291daa9d7b5") : secret "dns-default-metrics-tls" not found
Sep 09 16:09:24.149860 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:24.149825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:24.150559 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:24.150002    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:09:24.150559 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:24.150074    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls podName:19683a66-1e9f-4c96-85bf-5291daa9d7b5 nodeName:}" failed. No retries permitted until 2026-09-09 16:09:28.150054109 +0000 UTC m=+44.085672518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls") pod "dns-default-9s8wl" (UID: "19683a66-1e9f-4c96-85bf-5291daa9d7b5") : secret "dns-default-metrics-tls" not found
Sep 09 16:09:24.738259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:24.738222    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bmprr" event={"ID":"f38b903a-197a-466f-87d5-e3fac326978a","Type":"ContainerStarted","Data":"4f6e9aee83f4377ae41d1e3c55e27d0e123c538e4d8b2863981dc7d3dd2a847d"}
Sep 09 16:09:24.753172 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:24.753128    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bmprr" podStartSLOduration=2.054675107 podStartE2EDuration="4.753114242s" totalImagesPullingTime="2.698439135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:24.752351219 +0000 UTC m=+40.687969647" watchObservedRunningTime="2026-09-09 16:09:24.753114242 +0000 UTC m=+40.688732675"
Sep 09 16:09:28.173881 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:28.173842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:28.174340 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:28.173978    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:09:28.174340 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:28.174054    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls podName:19683a66-1e9f-4c96-85bf-5291daa9d7b5 nodeName:}" failed. No retries permitted until 2026-09-09 16:09:36.174037586 +0000 UTC m=+52.109655992 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls") pod "dns-default-9s8wl" (UID: "19683a66-1e9f-4c96-85bf-5291daa9d7b5") : secret "dns-default-metrics-tls" not found
Sep 09 16:09:34.514669 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.514637    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"]
Sep 09 16:09:34.527456 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.527429    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"]
Sep 09 16:09:34.527592 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.527527    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:09:34.529794 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.529763    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 16:09:34.530921 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.530905    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 16:09:34.531038 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.530907    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 16:09:34.531038 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.530938    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 16:09:34.545914 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.545892    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"]
Sep 09 16:09:34.565394 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.565373    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.571334 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.568429    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 16:09:34.571334 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.568602    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 16:09:34.571334 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.568668    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 16:09:34.571334 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.569097    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 16:09:34.572279 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.572260    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"]
Sep 09 16:09:34.613641 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.613621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.613776 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.613648    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.613776 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.613720    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-ca\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.613776 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.613738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-hub\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.613776 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.613755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71bc1392-2731-4cc5-9d41-631b6c5ea409-klusterlet-config\") pod \"klusterlet-addon-workmgr-76f4cb6b69-7mlcs\" (UID: \"71bc1392-2731-4cc5-9d41-631b6c5ea409\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:09:34.613776 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.613777    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqh6g\" (UniqueName: \"kubernetes.io/projected/71bc1392-2731-4cc5-9d41-631b6c5ea409-kube-api-access-lqh6g\") pod \"klusterlet-addon-workmgr-76f4cb6b69-7mlcs\" (UID: \"71bc1392-2731-4cc5-9d41-631b6c5ea409\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:09:34.613969 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.613796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q58xm\" (UniqueName: \"kubernetes.io/projected/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-kube-api-access-q58xm\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.613969 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.613811    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71bc1392-2731-4cc5-9d41-631b6c5ea409-tmp\") pod \"klusterlet-addon-workmgr-76f4cb6b69-7mlcs\" (UID: \"71bc1392-2731-4cc5-9d41-631b6c5ea409\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:09:34.613969 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.613825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.714281 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.714255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-hub\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.714356 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.714287    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71bc1392-2731-4cc5-9d41-631b6c5ea409-klusterlet-config\") pod \"klusterlet-addon-workmgr-76f4cb6b69-7mlcs\" (UID: \"71bc1392-2731-4cc5-9d41-631b6c5ea409\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:09:34.714356 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.714310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqh6g\" (UniqueName: \"kubernetes.io/projected/71bc1392-2731-4cc5-9d41-631b6c5ea409-kube-api-access-lqh6g\") pod \"klusterlet-addon-workmgr-76f4cb6b69-7mlcs\" (UID: \"71bc1392-2731-4cc5-9d41-631b6c5ea409\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:09:34.714451 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.714413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58xm\" (UniqueName: \"kubernetes.io/projected/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-kube-api-access-q58xm\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.714451 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.714443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71bc1392-2731-4cc5-9d41-631b6c5ea409-tmp\") pod \"klusterlet-addon-workmgr-76f4cb6b69-7mlcs\" (UID: \"71bc1392-2731-4cc5-9d41-631b6c5ea409\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:09:34.714552 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.714459    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.714552 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.714487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.714552 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.714505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.714666 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.714580    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-ca\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.714971 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.714923    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71bc1392-2731-4cc5-9d41-631b6c5ea409-tmp\") pod \"klusterlet-addon-workmgr-76f4cb6b69-7mlcs\" (UID: \"71bc1392-2731-4cc5-9d41-631b6c5ea409\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:09:34.715300 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.715281    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.717980 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.717950    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-ca\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.718064 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.717977    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.718064 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.717994    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-hub\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.718064 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.717960    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.718064 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.718029    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71bc1392-2731-4cc5-9d41-631b6c5ea409-klusterlet-config\") pod \"klusterlet-addon-workmgr-76f4cb6b69-7mlcs\" (UID: \"71bc1392-2731-4cc5-9d41-631b6c5ea409\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:09:34.726622 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.726599    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqh6g\" (UniqueName: \"kubernetes.io/projected/71bc1392-2731-4cc5-9d41-631b6c5ea409-kube-api-access-lqh6g\") pod \"klusterlet-addon-workmgr-76f4cb6b69-7mlcs\" (UID: \"71bc1392-2731-4cc5-9d41-631b6c5ea409\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:09:34.726785 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.726770    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q58xm\" (UniqueName: \"kubernetes.io/projected/4b6cebbf-2824-45dd-a453-2f3aa542a3cd-kube-api-access-q58xm\") pod \"cluster-proxy-proxy-agent-5966f759c9-65qw7\" (UID: \"4b6cebbf-2824-45dd-a453-2f3aa542a3cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:34.837642 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.837582    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:09:34.893072 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:34.893041    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:09:35.134150 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:35.134078    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"]
Sep 09 16:09:35.137229 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:09:35.137196    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bc1392_2731_4cc5_9d41_631b6c5ea409.slice/crio-c125a7f3e9f21b720c3f9e23c15cd975120fe8b11067f46694feb85dea8aa265 WatchSource:0}: Error finding container c125a7f3e9f21b720c3f9e23c15cd975120fe8b11067f46694feb85dea8aa265: Status 404 returned error can't find the container with id c125a7f3e9f21b720c3f9e23c15cd975120fe8b11067f46694feb85dea8aa265
Sep 09 16:09:35.187672 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:35.187645    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"]
Sep 09 16:09:35.190206 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:09:35.190181    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b6cebbf_2824_45dd_a453_2f3aa542a3cd.slice/crio-22cc00eb05a56c595010877bfc7cb369f4abed2a58d33b455a49cc0eb2f47b24 WatchSource:0}: Error finding container 22cc00eb05a56c595010877bfc7cb369f4abed2a58d33b455a49cc0eb2f47b24: Status 404 returned error can't find the container with id 22cc00eb05a56c595010877bfc7cb369f4abed2a58d33b455a49cc0eb2f47b24
Sep 09 16:09:35.766739 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:35.766690    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7" event={"ID":"4b6cebbf-2824-45dd-a453-2f3aa542a3cd","Type":"ContainerStarted","Data":"22cc00eb05a56c595010877bfc7cb369f4abed2a58d33b455a49cc0eb2f47b24"}
Sep 09 16:09:35.767809 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:35.767787    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs" event={"ID":"71bc1392-2731-4cc5-9d41-631b6c5ea409","Type":"ContainerStarted","Data":"c125a7f3e9f21b720c3f9e23c15cd975120fe8b11067f46694feb85dea8aa265"}
Sep 09 16:09:36.226186 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:36.226149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:36.226358 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:36.226323    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:09:36.226424 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:36.226400    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls podName:19683a66-1e9f-4c96-85bf-5291daa9d7b5 nodeName:}" failed. No retries permitted until 2026-09-09 16:09:52.226379465 +0000 UTC m=+68.161997858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls") pod "dns-default-9s8wl" (UID: "19683a66-1e9f-4c96-85bf-5291daa9d7b5") : secret "dns-default-metrics-tls" not found
Sep 09 16:09:41.786271 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:41.786185    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7" event={"ID":"4b6cebbf-2824-45dd-a453-2f3aa542a3cd","Type":"ContainerStarted","Data":"6fa8396b666c6135a7eb19fad1e8f7b6c9a94864946b983e534a898fe5ca4d3b"}
Sep 09 16:09:41.786271 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:41.786241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7" event={"ID":"4b6cebbf-2824-45dd-a453-2f3aa542a3cd","Type":"ContainerStarted","Data":"e3a05682aa35782a39824df13a9bf291d6052565efad0e05bc7f82b0371c7427"}
Sep 09 16:09:41.786271 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:41.786261    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7" event={"ID":"4b6cebbf-2824-45dd-a453-2f3aa542a3cd","Type":"ContainerStarted","Data":"d5ba0fc88c2fd4b5100c79e9326dff1e7d9b8f22e856f4e20b33cbd629b9b744"}
Sep 09 16:09:41.787670 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:41.787645    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs" event={"ID":"71bc1392-2731-4cc5-9d41-631b6c5ea409","Type":"ContainerStarted","Data":"0d039f2f01e677cb307364d04eb7ca80ef2b73d3508f45322caa18941c9669cb"}
Sep 09 16:09:41.787821 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:41.787806    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:09:41.789371 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:41.789345    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:09:41.810362 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:41.810319    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7" podStartSLOduration=1.465827171 podStartE2EDuration="7.810306045s" totalImagesPullingTime="6.344478874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:41.810076482 +0000 UTC m=+57.745694902" watchObservedRunningTime="2026-09-09 16:09:41.810306045 +0000 UTC m=+57.745924460"
Sep 09 16:09:41.827944 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:41.827904    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs" podStartSLOduration=1.4296189240000001 podStartE2EDuration="7.827891374s" totalImagesPullingTime="6.39827245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:41.82708232 +0000 UTC m=+57.762700745" watchObservedRunningTime="2026-09-09 16:09:41.827891374 +0000 UTC m=+57.763509790"
Sep 09 16:09:45.717299 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:45.717267    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-td8ml"
Sep 09 16:09:50.233714 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.233661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:50.236283 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.236265    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 16:09:50.246340 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.246311    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27913e87-038f-437f-a49e-6e3768968886-original-pull-secret\") pod \"global-pull-secret-syncer-psbxz\" (UID: \"27913e87-038f-437f-a49e-6e3768968886\") " pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:50.256436 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.256409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbxz"
Sep 09 16:09:50.334502 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.334471    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:09:50.337201 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.337183    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 16:09:50.344981 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:50.344964    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 16:09:50.345039 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:50.345034    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs podName:03652816-9d0a-45d3-9390-04cc6acc8cee nodeName:}" failed. No retries permitted until 2026-09-09 16:10:54.345012841 +0000 UTC m=+130.280631249 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs") pod "network-metrics-daemon-nwfs2" (UID: "03652816-9d0a-45d3-9390-04cc6acc8cee") : secret "metrics-daemon-secret" not found
Sep 09 16:09:50.435381 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.435359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgp6\" (UniqueName: \"kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6\") pod \"network-check-target-8njz2\" (UID: \"af80a467-9aa6-4c3b-81fd-d05dc3f09e75\") " pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:50.437772 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.437756    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 16:09:50.447706 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.447685    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 16:09:50.458751 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.458728    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcgp6\" (UniqueName: \"kubernetes.io/projected/af80a467-9aa6-4c3b-81fd-d05dc3f09e75-kube-api-access-zcgp6\") pod \"network-check-target-8njz2\" (UID: \"af80a467-9aa6-4c3b-81fd-d05dc3f09e75\") " pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:50.522427 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.522359    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-psbxz"]
Sep 09 16:09:50.527494 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:09:50.527466    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27913e87_038f_437f_a49e_6e3768968886.slice/crio-08c40773a77825afabf3c606b4994d26f98b0f4bfa7068d217ec10419dbf45ab WatchSource:0}: Error finding container 08c40773a77825afabf3c606b4994d26f98b0f4bfa7068d217ec10419dbf45ab: Status 404 returned error can't find the container with id 08c40773a77825afabf3c606b4994d26f98b0f4bfa7068d217ec10419dbf45ab
Sep 09 16:09:50.563582 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.563553    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-trsmq\""
Sep 09 16:09:50.571562 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.571541    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:50.811828 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.811744    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-psbxz" event={"ID":"27913e87-038f-437f-a49e-6e3768968886","Type":"ContainerStarted","Data":"08c40773a77825afabf3c606b4994d26f98b0f4bfa7068d217ec10419dbf45ab"}
Sep 09 16:09:50.835353 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:50.835323    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8njz2"]
Sep 09 16:09:50.838092 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:09:50.838068    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf80a467_9aa6_4c3b_81fd_d05dc3f09e75.slice/crio-47c09f41c50f0e21eba4113b3a0e551838fb6fc722bfa58d542a24e08fd32884 WatchSource:0}: Error finding container 47c09f41c50f0e21eba4113b3a0e551838fb6fc722bfa58d542a24e08fd32884: Status 404 returned error can't find the container with id 47c09f41c50f0e21eba4113b3a0e551838fb6fc722bfa58d542a24e08fd32884
Sep 09 16:09:51.816019 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:51.815984    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8njz2" event={"ID":"af80a467-9aa6-4c3b-81fd-d05dc3f09e75","Type":"ContainerStarted","Data":"47c09f41c50f0e21eba4113b3a0e551838fb6fc722bfa58d542a24e08fd32884"}
Sep 09 16:09:52.247966 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:52.247924    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:09:52.248224 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:52.248083    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:09:52.248224 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:09:52.248152    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls podName:19683a66-1e9f-4c96-85bf-5291daa9d7b5 nodeName:}" failed. No retries permitted until 2026-09-09 16:10:24.248135886 +0000 UTC m=+100.183754284 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls") pod "dns-default-9s8wl" (UID: "19683a66-1e9f-4c96-85bf-5291daa9d7b5") : secret "dns-default-metrics-tls" not found
Sep 09 16:09:55.829824 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:55.829787    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-psbxz" event={"ID":"27913e87-038f-437f-a49e-6e3768968886","Type":"ContainerStarted","Data":"9d4c530726b04cca7ef2b45dfe9dea08edf36879b62d3f6e6ba9eaeec2ff3b14"}
Sep 09 16:09:55.831279 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:55.831254    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8njz2" event={"ID":"af80a467-9aa6-4c3b-81fd-d05dc3f09e75","Type":"ContainerStarted","Data":"6bc5dfcb997aa58c0c9ca52fd9dd5c46f21c40dc9addb121673abc0d5aaa868f"}
Sep 09 16:09:55.831402 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:55.831391    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:09:55.844494 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:55.844455    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-psbxz" podStartSLOduration=66.768009955 podStartE2EDuration="1m11.844442504s" totalImagesPullingTime="5.076432549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:55.843727959 +0000 UTC m=+71.779346389" watchObservedRunningTime="2026-09-09 16:09:55.844442504 +0000 UTC m=+71.780060920"
Sep 09 16:09:55.857848 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:09:55.857771    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8njz2" podStartSLOduration=67.090263727 podStartE2EDuration="1m11.857758773s" totalImagesPullingTime="4.767495046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:55.857591067 +0000 UTC m=+71.793209501" watchObservedRunningTime="2026-09-09 16:09:55.857758773 +0000 UTC m=+71.793377189"
Sep 09 16:10:24.257686 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:10:24.257649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:10:24.258086 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:10:24.257820    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:10:24.258086 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:10:24.257898    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls podName:19683a66-1e9f-4c96-85bf-5291daa9d7b5 nodeName:}" failed. No retries permitted until 2026-09-09 16:11:28.257878197 +0000 UTC m=+164.193496610 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls") pod "dns-default-9s8wl" (UID: "19683a66-1e9f-4c96-85bf-5291daa9d7b5") : secret "dns-default-metrics-tls" not found
Sep 09 16:10:26.837184 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:10:26.837153    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8njz2"
Sep 09 16:10:54.364650 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:10:54.364611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:10:54.365046 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:10:54.364759    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 16:10:54.365046 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:10:54.364827    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs podName:03652816-9d0a-45d3-9390-04cc6acc8cee nodeName:}" failed. No retries permitted until 2026-09-09 16:12:56.364811161 +0000 UTC m=+252.300429568 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs") pod "network-metrics-daemon-nwfs2" (UID: "03652816-9d0a-45d3-9390-04cc6acc8cee") : secret "metrics-daemon-secret" not found
Sep 09 16:11:23.430360 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:11:23.430312    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9s8wl" podUID="19683a66-1e9f-4c96-85bf-5291daa9d7b5"
Sep 09 16:11:23.566256 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:11:23.566207    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:11:24.077879 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:11:24.077851    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:11:28.283461 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:11:28.283425    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:11:28.283858 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:11:28.283544    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:11:28.283858 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:11:28.283599    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls podName:19683a66-1e9f-4c96-85bf-5291daa9d7b5 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:30.283583328 +0000 UTC m=+286.219201743 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls") pod "dns-default-9s8wl" (UID: "19683a66-1e9f-4c96-85bf-5291daa9d7b5") : secret "dns-default-metrics-tls" not found
Sep 09 16:11:38.548064 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:11:38.547968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:11:41.788924 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:11:41.788879    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs" podUID="71bc1392-2731-4cc5-9d41-631b6c5ea409" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused"
Sep 09 16:11:42.128144 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:11:42.128114    2576 generic.go:356] "Generic (PLEG): container finished" podID="71bc1392-2731-4cc5-9d41-631b6c5ea409" containerID="0d039f2f01e677cb307364d04eb7ca80ef2b73d3508f45322caa18941c9669cb" exitCode=1
Sep 09 16:11:42.128276 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:11:42.128178    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs" event={"ID":"71bc1392-2731-4cc5-9d41-631b6c5ea409","Type":"ContainerDied","Data":"0d039f2f01e677cb307364d04eb7ca80ef2b73d3508f45322caa18941c9669cb"}
Sep 09 16:11:42.128496 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:11:42.128461    2576 scope.go:117] "RemoveContainer" containerID="0d039f2f01e677cb307364d04eb7ca80ef2b73d3508f45322caa18941c9669cb"
Sep 09 16:11:43.132937 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:11:43.132901    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs" event={"ID":"71bc1392-2731-4cc5-9d41-631b6c5ea409","Type":"ContainerStarted","Data":"0eed010f60a218fcf7b8da269f85bf2f1fd04144cd97fe881b408942c51b191a"}
Sep 09 16:11:43.133392 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:11:43.133189    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:11:43.133844 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:11:43.133824    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f4cb6b69-7mlcs"
Sep 09 16:12:56.384735 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:56.384658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:12:56.385237 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:12:56.384808    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 16:12:56.385237 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:12:56.384882    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs podName:03652816-9d0a-45d3-9390-04cc6acc8cee nodeName:}" failed. No retries permitted until 2026-09-09 16:14:58.384866234 +0000 UTC m=+374.320484628 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs") pod "network-metrics-daemon-nwfs2" (UID: "03652816-9d0a-45d3-9390-04cc6acc8cee") : secret "metrics-daemon-secret" not found
Sep 09 16:12:57.190642 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.190604    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lmssw"]
Sep 09 16:12:57.192843 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.192828    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.195270 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.195238    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 16:12:57.195372 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.195331    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 16:12:57.195427 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.195332    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 16:12:57.195594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.195580    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 16:12:57.196379 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.196365    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rd78t\""
Sep 09 16:12:57.204946 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.204926    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lmssw"]
Sep 09 16:12:57.291998 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.291958    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqpxb\" (UniqueName: \"kubernetes.io/projected/52d36234-dac2-4c5c-822c-aad4d084b857-kube-api-access-gqpxb\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.292164 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.292055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52d36234-dac2-4c5c-822c-aad4d084b857-crio-socket\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.292164 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.292083    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52d36234-dac2-4c5c-822c-aad4d084b857-data-volume\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.292164 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.292125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.292261 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.292214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.393406 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.393364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.393406 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.393416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqpxb\" (UniqueName: \"kubernetes.io/projected/52d36234-dac2-4c5c-822c-aad4d084b857-kube-api-access-gqpxb\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.393915 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.393485    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52d36234-dac2-4c5c-822c-aad4d084b857-crio-socket\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.393915 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.393510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52d36234-dac2-4c5c-822c-aad4d084b857-data-volume\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.393915 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:12:57.393523    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:12:57.393915 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:12:57.393594    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls podName:52d36234-dac2-4c5c-822c-aad4d084b857 nodeName:}" failed. No retries permitted until 2026-09-09 16:12:57.893575357 +0000 UTC m=+253.829193751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lmssw" (UID: "52d36234-dac2-4c5c-822c-aad4d084b857") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:12:57.393915 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.393600    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52d36234-dac2-4c5c-822c-aad4d084b857-crio-socket\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.393915 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.393653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.393915 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:12:57.393756    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:12:57.393915 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:12:57.393814    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm podName:52d36234-dac2-4c5c-822c-aad4d084b857 nodeName:}" failed. No retries permitted until 2026-09-09 16:12:57.893796787 +0000 UTC m=+253.829415195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lmssw" (UID: "52d36234-dac2-4c5c-822c-aad4d084b857") : configmap "kube-rbac-proxy" not found
Sep 09 16:12:57.393915 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.393847    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52d36234-dac2-4c5c-822c-aad4d084b857-data-volume\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.401658 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.401638    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqpxb\" (UniqueName: \"kubernetes.io/projected/52d36234-dac2-4c5c-822c-aad4d084b857-kube-api-access-gqpxb\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.898424 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.898383    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.898424 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:57.898431    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:57.898632 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:12:57.898535    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:12:57.898632 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:12:57.898548    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:12:57.898632 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:12:57.898602    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls podName:52d36234-dac2-4c5c-822c-aad4d084b857 nodeName:}" failed. No retries permitted until 2026-09-09 16:12:58.898587806 +0000 UTC m=+254.834206201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lmssw" (UID: "52d36234-dac2-4c5c-822c-aad4d084b857") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:12:57.898632 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:12:57.898621    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm podName:52d36234-dac2-4c5c-822c-aad4d084b857 nodeName:}" failed. No retries permitted until 2026-09-09 16:12:58.89860652 +0000 UTC m=+254.834224914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lmssw" (UID: "52d36234-dac2-4c5c-822c-aad4d084b857") : configmap "kube-rbac-proxy" not found
Sep 09 16:12:58.905208 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:58.905167    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:58.905592 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:58.905260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:12:58.905592 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:12:58.905314    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:12:58.905592 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:12:58.905345    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:12:58.905592 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:12:58.905379    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls podName:52d36234-dac2-4c5c-822c-aad4d084b857 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:00.905363852 +0000 UTC m=+256.840982246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lmssw" (UID: "52d36234-dac2-4c5c-822c-aad4d084b857") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:12:58.905592 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:12:58.905407    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm podName:52d36234-dac2-4c5c-822c-aad4d084b857 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:00.905393726 +0000 UTC m=+256.841012120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lmssw" (UID: "52d36234-dac2-4c5c-822c-aad4d084b857") : configmap "kube-rbac-proxy" not found
Sep 09 16:12:59.744879 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:12:59.744852    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5fhrb_0e49d49c-6384-4195-8385-374e3552fc45/dns-node-resolver/0.log"
Sep 09 16:13:00.920649 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:00.920612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:13:00.921023 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:00.920728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:13:00.921023 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:13:00.920788    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:13:00.921023 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:13:00.920789    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:13:00.921023 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:13:00.920854    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm podName:52d36234-dac2-4c5c-822c-aad4d084b857 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:04.920837339 +0000 UTC m=+260.856455733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lmssw" (UID: "52d36234-dac2-4c5c-822c-aad4d084b857") : configmap "kube-rbac-proxy" not found
Sep 09 16:13:00.921023 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:13:00.920867    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls podName:52d36234-dac2-4c5c-822c-aad4d084b857 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:04.920861628 +0000 UTC m=+260.856480022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lmssw" (UID: "52d36234-dac2-4c5c-822c-aad4d084b857") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:13:00.946082 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:00.946061    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tlk4z_96e4e031-e5c2-46a3-a044-0b0d6da36fb1/node-ca/0.log"
Sep 09 16:13:01.745884 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:01.745854    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bmprr_f38b903a-197a-466f-87d5-e3fac326978a/serve-healthcheck-canary/0.log"
Sep 09 16:13:04.946414 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:04.946334    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:13:04.946414 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:04.946407    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:13:04.946886 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:13:04.946477    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:13:04.946886 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:13:04.946479    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:13:04.946886 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:13:04.946534    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm podName:52d36234-dac2-4c5c-822c-aad4d084b857 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:12.946518394 +0000 UTC m=+268.882136790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lmssw" (UID: "52d36234-dac2-4c5c-822c-aad4d084b857") : configmap "kube-rbac-proxy" not found
Sep 09 16:13:04.946886 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:13:04.946547    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls podName:52d36234-dac2-4c5c-822c-aad4d084b857 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:12.946541297 +0000 UTC m=+268.882159690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lmssw" (UID: "52d36234-dac2-4c5c-822c-aad4d084b857") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:13:13.001461 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:13.001357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:13:13.002037 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:13.001475    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:13:13.002194 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:13.002172    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52d36234-dac2-4c5c-822c-aad4d084b857-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:13:13.003851 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:13.003832    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52d36234-dac2-4c5c-822c-aad4d084b857-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmssw\" (UID: \"52d36234-dac2-4c5c-822c-aad4d084b857\") " pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:13:13.101110 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:13.101062    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lmssw"
Sep 09 16:13:13.367918 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:13.367849    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lmssw"]
Sep 09 16:13:13.371091 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:13:13.371057    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d36234_dac2_4c5c_822c_aad4d084b857.slice/crio-22d636af54c24913e3aae7d020f3422cbc87e521d6223b4eebb36b3e33442006 WatchSource:0}: Error finding container 22d636af54c24913e3aae7d020f3422cbc87e521d6223b4eebb36b3e33442006: Status 404 returned error can't find the container with id 22d636af54c24913e3aae7d020f3422cbc87e521d6223b4eebb36b3e33442006
Sep 09 16:13:13.382993 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:13.382966    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lmssw" event={"ID":"52d36234-dac2-4c5c-822c-aad4d084b857","Type":"ContainerStarted","Data":"22d636af54c24913e3aae7d020f3422cbc87e521d6223b4eebb36b3e33442006"}
Sep 09 16:13:14.387439 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:14.387358    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lmssw" event={"ID":"52d36234-dac2-4c5c-822c-aad4d084b857","Type":"ContainerStarted","Data":"891deaaf8d9701bcd5cd6b3b7b92bb2e8f7bf526a0d236196bd0ec55435533a2"}
Sep 09 16:13:14.387439 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:14.387418    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lmssw" event={"ID":"52d36234-dac2-4c5c-822c-aad4d084b857","Type":"ContainerStarted","Data":"ded9135657a2f9839471f4663e2280c565b45048d4f7353c5c1a4f1e5b2b8736"}
Sep 09 16:13:16.395832 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:16.395800    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lmssw" event={"ID":"52d36234-dac2-4c5c-822c-aad4d084b857","Type":"ContainerStarted","Data":"e960587bd6288fd8cb136e2996a00c1d051eeeca6ab0a23be455c62681e2fb53"}
Sep 09 16:13:16.412566 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:16.412519    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lmssw" podStartSLOduration=17.155286073 podStartE2EDuration="19.412503177s" totalImagesPullingTime="2.257217104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:12:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:13:16.412480415 +0000 UTC m=+272.348098842" watchObservedRunningTime="2026-09-09 16:13:16.412503177 +0000 UTC m=+272.348121592"
Sep 09 16:13:20.827892 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.827859    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fb76c6d86-lvxfv"]
Sep 09 16:13:20.830798 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.830774    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.833169 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.833143    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6jv6x\""
Sep 09 16:13:20.833276 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.833176    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 16:13:20.834087 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.834068    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 16:13:20.834188 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.834105    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 16:13:20.837997 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.837767    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 16:13:20.842658 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.842636    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fb76c6d86-lvxfv"]
Sep 09 16:13:20.855825 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.855801    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/15c10237-6222-4e3d-bc36-6ae7eac9019e-ca-trust-extracted\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.855914 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.855831    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15c10237-6222-4e3d-bc36-6ae7eac9019e-trusted-ca\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.855914 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.855858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkxpr\" (UniqueName: \"kubernetes.io/projected/15c10237-6222-4e3d-bc36-6ae7eac9019e-kube-api-access-vkxpr\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.856010 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.855937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/15c10237-6222-4e3d-bc36-6ae7eac9019e-image-registry-private-configuration\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.856010 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.855960    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15c10237-6222-4e3d-bc36-6ae7eac9019e-registry-tls\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.856010 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.855982    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/15c10237-6222-4e3d-bc36-6ae7eac9019e-registry-certificates\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.856142 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.856013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15c10237-6222-4e3d-bc36-6ae7eac9019e-bound-sa-token\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.856142 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.856051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/15c10237-6222-4e3d-bc36-6ae7eac9019e-installation-pull-secrets\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.957171 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.957138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/15c10237-6222-4e3d-bc36-6ae7eac9019e-image-registry-private-configuration\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.957171 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.957173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15c10237-6222-4e3d-bc36-6ae7eac9019e-registry-tls\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.957395 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.957200    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/15c10237-6222-4e3d-bc36-6ae7eac9019e-registry-certificates\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.957395 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.957243    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15c10237-6222-4e3d-bc36-6ae7eac9019e-bound-sa-token\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.957395 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.957291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/15c10237-6222-4e3d-bc36-6ae7eac9019e-installation-pull-secrets\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.957395 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.957316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/15c10237-6222-4e3d-bc36-6ae7eac9019e-ca-trust-extracted\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.957581 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.957429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15c10237-6222-4e3d-bc36-6ae7eac9019e-trusted-ca\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.957581 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.957459    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkxpr\" (UniqueName: \"kubernetes.io/projected/15c10237-6222-4e3d-bc36-6ae7eac9019e-kube-api-access-vkxpr\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.957974 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.957953    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/15c10237-6222-4e3d-bc36-6ae7eac9019e-ca-trust-extracted\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.958198 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.958170    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/15c10237-6222-4e3d-bc36-6ae7eac9019e-registry-certificates\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.958378 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.958357    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15c10237-6222-4e3d-bc36-6ae7eac9019e-trusted-ca\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.959718 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.959682    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/15c10237-6222-4e3d-bc36-6ae7eac9019e-installation-pull-secrets\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.960159 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.960137    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15c10237-6222-4e3d-bc36-6ae7eac9019e-registry-tls\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.960295 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.960276    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/15c10237-6222-4e3d-bc36-6ae7eac9019e-image-registry-private-configuration\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.964925 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.964907    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15c10237-6222-4e3d-bc36-6ae7eac9019e-bound-sa-token\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:20.966338 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:20.966320    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkxpr\" (UniqueName: \"kubernetes.io/projected/15c10237-6222-4e3d-bc36-6ae7eac9019e-kube-api-access-vkxpr\") pod \"image-registry-5fb76c6d86-lvxfv\" (UID: \"15c10237-6222-4e3d-bc36-6ae7eac9019e\") " pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:21.140408 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:21.140327    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:21.415012 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:21.414939    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fb76c6d86-lvxfv"]
Sep 09 16:13:21.417508 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:13:21.417468    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c10237_6222_4e3d_bc36_6ae7eac9019e.slice/crio-a707387bc7825fbe7a4b3d5c07f7e36c6441a8303110660dbef8e1a9d37251fc WatchSource:0}: Error finding container a707387bc7825fbe7a4b3d5c07f7e36c6441a8303110660dbef8e1a9d37251fc: Status 404 returned error can't find the container with id a707387bc7825fbe7a4b3d5c07f7e36c6441a8303110660dbef8e1a9d37251fc
Sep 09 16:13:22.414616 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:22.414584    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv" event={"ID":"15c10237-6222-4e3d-bc36-6ae7eac9019e","Type":"ContainerStarted","Data":"dc7e40857e33c2c91375632807957cf6de189a9de8b3464babef552aef2dbda7"}
Sep 09 16:13:22.415008 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:22.414630    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv" event={"ID":"15c10237-6222-4e3d-bc36-6ae7eac9019e","Type":"ContainerStarted","Data":"a707387bc7825fbe7a4b3d5c07f7e36c6441a8303110660dbef8e1a9d37251fc"}
Sep 09 16:13:22.415008 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:22.414726    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:22.433549 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:22.433496    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv" podStartSLOduration=2.433483451 podStartE2EDuration="2.433483451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:13:22.432253028 +0000 UTC m=+278.367871463" watchObservedRunningTime="2026-09-09 16:13:22.433483451 +0000 UTC m=+278.369101867"
Sep 09 16:13:27.078516 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:13:27.078457    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9s8wl" podUID="19683a66-1e9f-4c96-85bf-5291daa9d7b5"
Sep 09 16:13:27.429245 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:27.429170    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:13:30.318463 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:30.318423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:13:30.320885 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:30.320862    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19683a66-1e9f-4c96-85bf-5291daa9d7b5-metrics-tls\") pod \"dns-default-9s8wl\" (UID: \"19683a66-1e9f-4c96-85bf-5291daa9d7b5\") " pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:13:30.432687 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:30.432659    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pgn4p\""
Sep 09 16:13:30.440784 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:30.440761    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:13:30.711312 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:30.711243    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9s8wl"]
Sep 09 16:13:30.714323 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:13:30.714298    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19683a66_1e9f_4c96_85bf_5291daa9d7b5.slice/crio-808867e5edca0df8d5f38bf780e355c903636ebc5027d61330d7d69e69e9594a WatchSource:0}: Error finding container 808867e5edca0df8d5f38bf780e355c903636ebc5027d61330d7d69e69e9594a: Status 404 returned error can't find the container with id 808867e5edca0df8d5f38bf780e355c903636ebc5027d61330d7d69e69e9594a
Sep 09 16:13:31.442290 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:31.442254    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9s8wl" event={"ID":"19683a66-1e9f-4c96-85bf-5291daa9d7b5","Type":"ContainerStarted","Data":"808867e5edca0df8d5f38bf780e355c903636ebc5027d61330d7d69e69e9594a"}
Sep 09 16:13:32.446894 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:32.446811    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9s8wl" event={"ID":"19683a66-1e9f-4c96-85bf-5291daa9d7b5","Type":"ContainerStarted","Data":"269d9ae85bcacb983ed49370fc26b9c6bd603259f02f381fb26dfcb60406fbe3"}
Sep 09 16:13:32.446894 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:32.446866    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9s8wl" event={"ID":"19683a66-1e9f-4c96-85bf-5291daa9d7b5","Type":"ContainerStarted","Data":"1d4c89ce3e54115f27e1667aa51d96efef1496e085ec0e8d108aa24883610cfe"}
Sep 09 16:13:32.447368 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:32.446902    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:13:32.463453 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:32.463406    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9s8wl" podStartSLOduration=250.987365702 podStartE2EDuration="4m12.46339251s" totalImagesPullingTime="1.476026808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:13:32.463105466 +0000 UTC m=+288.398723901" watchObservedRunningTime="2026-09-09 16:13:32.46339251 +0000 UTC m=+288.399010917"
Sep 09 16:13:37.362103 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.362070    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7gnf2"]
Sep 09 16:13:37.365017 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.365002    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.367798 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.367777    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 16:13:37.368151 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.368095    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 16:13:37.368151 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.368133    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9hbdg\""
Sep 09 16:13:37.368378 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.368356    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 16:13:37.368817 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.368801    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 16:13:37.369081 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.369068    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 16:13:37.369081 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.369074    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 16:13:37.470112 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.470080    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-accelerators-collector-config\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.470268 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.470119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-wtmp\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.470268 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.470140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.470268 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.470194    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86c73f0e-1792-412c-8fb8-294f4d6deaad-sys\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.470268 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.470262    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/86c73f0e-1792-412c-8fb8-294f4d6deaad-root\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.470425 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.470283    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-tls\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.470425 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.470305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-textfile\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.470425 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.470320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86c73f0e-1792-412c-8fb8-294f4d6deaad-metrics-client-ca\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.470425 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.470360    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9scsg\" (UniqueName: \"kubernetes.io/projected/86c73f0e-1792-412c-8fb8-294f4d6deaad-kube-api-access-9scsg\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.571656 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.571627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-tls\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.571656 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.571658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-textfile\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.571884 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.571675    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86c73f0e-1792-412c-8fb8-294f4d6deaad-metrics-client-ca\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.571884 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.571718    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9scsg\" (UniqueName: \"kubernetes.io/projected/86c73f0e-1792-412c-8fb8-294f4d6deaad-kube-api-access-9scsg\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.571884 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.571743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-accelerators-collector-config\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.571884 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.571772    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-wtmp\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.571884 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.571792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.571884 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:13:37.571799    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 16:13:37.571884 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.571818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86c73f0e-1792-412c-8fb8-294f4d6deaad-sys\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.571884 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:13:37.571872    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-tls podName:86c73f0e-1792-412c-8fb8-294f4d6deaad nodeName:}" failed. No retries permitted until 2026-09-09 16:13:38.071853543 +0000 UTC m=+294.007471938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-tls") pod "node-exporter-7gnf2" (UID: "86c73f0e-1792-412c-8fb8-294f4d6deaad") : secret "node-exporter-tls" not found
Sep 09 16:13:37.571884 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.571885    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86c73f0e-1792-412c-8fb8-294f4d6deaad-sys\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.572239 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.571951    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-wtmp\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.572239 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.571987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/86c73f0e-1792-412c-8fb8-294f4d6deaad-root\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.572239 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.572052    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-textfile\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.572239 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.572069    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/86c73f0e-1792-412c-8fb8-294f4d6deaad-root\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.572372 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.572285    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86c73f0e-1792-412c-8fb8-294f4d6deaad-metrics-client-ca\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.572372 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.572337    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-accelerators-collector-config\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.574229 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.574213    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:37.580830 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:37.580810    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9scsg\" (UniqueName: \"kubernetes.io/projected/86c73f0e-1792-412c-8fb8-294f4d6deaad-kube-api-access-9scsg\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:38.075029 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:38.074992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-tls\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:38.075189 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:13:38.075143    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 16:13:38.075230 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:13:38.075205    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-tls podName:86c73f0e-1792-412c-8fb8-294f4d6deaad nodeName:}" failed. No retries permitted until 2026-09-09 16:13:39.075190761 +0000 UTC m=+295.010809158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-tls") pod "node-exporter-7gnf2" (UID: "86c73f0e-1792-412c-8fb8-294f4d6deaad") : secret "node-exporter-tls" not found
Sep 09 16:13:39.084359 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:39.084312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-tls\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:39.086891 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:39.086867    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86c73f0e-1792-412c-8fb8-294f4d6deaad-node-exporter-tls\") pod \"node-exporter-7gnf2\" (UID: \"86c73f0e-1792-412c-8fb8-294f4d6deaad\") " pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:39.174093 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:39.174062    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7gnf2"
Sep 09 16:13:39.185057 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:13:39.185033    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c73f0e_1792_412c_8fb8_294f4d6deaad.slice/crio-395f004db2df1cce2c30c3f0418bec6aad9596eb6bc2590729b1b59a964e1859 WatchSource:0}: Error finding container 395f004db2df1cce2c30c3f0418bec6aad9596eb6bc2590729b1b59a964e1859: Status 404 returned error can't find the container with id 395f004db2df1cce2c30c3f0418bec6aad9596eb6bc2590729b1b59a964e1859
Sep 09 16:13:39.468496 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:39.468409    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gnf2" event={"ID":"86c73f0e-1792-412c-8fb8-294f4d6deaad","Type":"ContainerStarted","Data":"395f004db2df1cce2c30c3f0418bec6aad9596eb6bc2590729b1b59a964e1859"}
Sep 09 16:13:41.549164 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:13:41.549126    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nwfs2" podUID="03652816-9d0a-45d3-9390-04cc6acc8cee"
Sep 09 16:13:42.452216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:42.452184    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9s8wl"
Sep 09 16:13:43.422320 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:43.422293    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5fb76c6d86-lvxfv"
Sep 09 16:13:43.482107 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:43.482069    2576 generic.go:356] "Generic (PLEG): container finished" podID="86c73f0e-1792-412c-8fb8-294f4d6deaad" containerID="15faa3af9efbd63f5df6ad58a94cb033564e68dd168d8999e1bcffe2ae4db0e0" exitCode=0
Sep 09 16:13:43.482419 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:43.482111    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gnf2" event={"ID":"86c73f0e-1792-412c-8fb8-294f4d6deaad","Type":"ContainerDied","Data":"15faa3af9efbd63f5df6ad58a94cb033564e68dd168d8999e1bcffe2ae4db0e0"}
Sep 09 16:13:44.450377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:44.450343    2576 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 16:13:44.486684 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:44.486652    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gnf2" event={"ID":"86c73f0e-1792-412c-8fb8-294f4d6deaad","Type":"ContainerStarted","Data":"dfd1393c39845446efa36c4d6081cd76c475e82622e69784c45c8b684b60fc49"}
Sep 09 16:13:44.486863 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:44.486721    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gnf2" event={"ID":"86c73f0e-1792-412c-8fb8-294f4d6deaad","Type":"ContainerStarted","Data":"027d367e7de0170ae133ea264f315c7a9728d82428c5b6a71378b91f9f432ef8"}
Sep 09 16:13:44.507269 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:44.507112    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7gnf2" podStartSLOduration=3.75954821 podStartE2EDuration="7.507099047s" totalImagesPullingTime="3.679816013s" totalInitContainerRuntime="67.734824ms" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:13:44.506299106 +0000 UTC m=+300.441917573" watchObservedRunningTime="2026-09-09 16:13:44.507099047 +0000 UTC m=+300.442717463"
Sep 09 16:13:44.894489 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:44.894453    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7" podUID="4b6cebbf-2824-45dd-a453-2f3aa542a3cd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 16:13:54.548964 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:54.548923    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:13:54.894987 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:13:54.894912    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7" podUID="4b6cebbf-2824-45dd-a453-2f3aa542a3cd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 16:14:04.895005 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:04.894922    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7" podUID="4b6cebbf-2824-45dd-a453-2f3aa542a3cd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 16:14:04.895005 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:04.895002    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7"
Sep 09 16:14:04.895478 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:04.895448    2576 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6fa8396b666c6135a7eb19fad1e8f7b6c9a94864946b983e534a898fe5ca4d3b"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 09 16:14:04.895516 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:04.895498    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7" podUID="4b6cebbf-2824-45dd-a453-2f3aa542a3cd" containerName="service-proxy" containerID="cri-o://6fa8396b666c6135a7eb19fad1e8f7b6c9a94864946b983e534a898fe5ca4d3b" gracePeriod=30
Sep 09 16:14:05.552463 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:05.552428    2576 generic.go:356] "Generic (PLEG): container finished" podID="4b6cebbf-2824-45dd-a453-2f3aa542a3cd" containerID="6fa8396b666c6135a7eb19fad1e8f7b6c9a94864946b983e534a898fe5ca4d3b" exitCode=2
Sep 09 16:14:05.552624 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:05.552495    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7" event={"ID":"4b6cebbf-2824-45dd-a453-2f3aa542a3cd","Type":"ContainerDied","Data":"6fa8396b666c6135a7eb19fad1e8f7b6c9a94864946b983e534a898fe5ca4d3b"}
Sep 09 16:14:05.552624 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:05.552536    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5966f759c9-65qw7" event={"ID":"4b6cebbf-2824-45dd-a453-2f3aa542a3cd","Type":"ContainerStarted","Data":"bdad65e8e27336fc345e3288df82b2c4b37f6fc6c0cec324185e72c3ec734ab3"}
Sep 09 16:14:17.949653 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:17.949628    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bmprr_f38b903a-197a-466f-87d5-e3fac326978a/serve-healthcheck-canary/0.log"
Sep 09 16:14:58.448448 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:58.448352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:14:58.450868 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:58.450845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03652816-9d0a-45d3-9390-04cc6acc8cee-metrics-certs\") pod \"network-metrics-daemon-nwfs2\" (UID: \"03652816-9d0a-45d3-9390-04cc6acc8cee\") " pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:14:58.752907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:58.752882    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zj7bv\""
Sep 09 16:14:58.761181 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:58.761163    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwfs2"
Sep 09 16:14:59.028208 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:59.028185    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nwfs2"]
Sep 09 16:14:59.030872 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:14:59.030836    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03652816_9d0a_45d3_9390_04cc6acc8cee.slice/crio-0c9f83021ab831b8d20f45cbe29888c86252a044795d9bdfae83d86e00a22525 WatchSource:0}: Error finding container 0c9f83021ab831b8d20f45cbe29888c86252a044795d9bdfae83d86e00a22525: Status 404 returned error can't find the container with id 0c9f83021ab831b8d20f45cbe29888c86252a044795d9bdfae83d86e00a22525
Sep 09 16:14:59.033147 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:59.033129    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:14:59.715886 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:14:59.715842    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwfs2" event={"ID":"03652816-9d0a-45d3-9390-04cc6acc8cee","Type":"ContainerStarted","Data":"0c9f83021ab831b8d20f45cbe29888c86252a044795d9bdfae83d86e00a22525"}
Sep 09 16:15:00.721249 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:15:00.721213    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwfs2" event={"ID":"03652816-9d0a-45d3-9390-04cc6acc8cee","Type":"ContainerStarted","Data":"f3ba5d3a28af37b35ed111b694d719ab668858beef6d77beab77a6136b0f0d6b"}
Sep 09 16:15:00.721741 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:15:00.721275    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwfs2" event={"ID":"03652816-9d0a-45d3-9390-04cc6acc8cee","Type":"ContainerStarted","Data":"ed11bedd0b164ddc212479bd36b3c42285d937c1e1bba033502fa4c0d33b5f01"}
Sep 09 16:15:00.741053 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:15:00.741004    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nwfs2" podStartSLOduration=375.563898923 podStartE2EDuration="6m16.740988373s" totalImagesPullingTime="1.17708945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:00.739773677 +0000 UTC m=+376.675392104" watchObservedRunningTime="2026-09-09 16:15:00.740988373 +0000 UTC m=+376.676606790"
Sep 09 16:17:17.079432 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.079399    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"]
Sep 09 16:17:17.081957 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.081942    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:17.088300 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.088278    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 16:17:17.088409 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.088355    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 16:17:17.088409 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.088377    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 16:17:17.089213 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.089197    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 16:17:17.089262 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.089197    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 16:17:17.092818 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.092801    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-dwhmp\""
Sep 09 16:17:17.103896 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.103871    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"]
Sep 09 16:17:17.187922 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.187891    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6446a118-60ed-4da3-9f5b-1adfc9851694-cert\") pod \"lws-controller-manager-7bc87557cd-pc7zb\" (UID: \"6446a118-60ed-4da3-9f5b-1adfc9851694\") " pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:17.187922 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.187927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6446a118-60ed-4da3-9f5b-1adfc9851694-metrics-cert\") pod \"lws-controller-manager-7bc87557cd-pc7zb\" (UID: \"6446a118-60ed-4da3-9f5b-1adfc9851694\") " pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:17.188156 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.187973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjzzv\" (UniqueName: \"kubernetes.io/projected/6446a118-60ed-4da3-9f5b-1adfc9851694-kube-api-access-hjzzv\") pod \"lws-controller-manager-7bc87557cd-pc7zb\" (UID: \"6446a118-60ed-4da3-9f5b-1adfc9851694\") " pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:17.188156 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.188004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6446a118-60ed-4da3-9f5b-1adfc9851694-manager-config\") pod \"lws-controller-manager-7bc87557cd-pc7zb\" (UID: \"6446a118-60ed-4da3-9f5b-1adfc9851694\") " pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:17.289214 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.289183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6446a118-60ed-4da3-9f5b-1adfc9851694-cert\") pod \"lws-controller-manager-7bc87557cd-pc7zb\" (UID: \"6446a118-60ed-4da3-9f5b-1adfc9851694\") " pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:17.289317 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.289217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6446a118-60ed-4da3-9f5b-1adfc9851694-metrics-cert\") pod \"lws-controller-manager-7bc87557cd-pc7zb\" (UID: \"6446a118-60ed-4da3-9f5b-1adfc9851694\") " pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:17.289317 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.289264    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjzzv\" (UniqueName: \"kubernetes.io/projected/6446a118-60ed-4da3-9f5b-1adfc9851694-kube-api-access-hjzzv\") pod \"lws-controller-manager-7bc87557cd-pc7zb\" (UID: \"6446a118-60ed-4da3-9f5b-1adfc9851694\") " pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:17.289317 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.289294    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6446a118-60ed-4da3-9f5b-1adfc9851694-manager-config\") pod \"lws-controller-manager-7bc87557cd-pc7zb\" (UID: \"6446a118-60ed-4da3-9f5b-1adfc9851694\") " pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:17.290005 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.289986    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6446a118-60ed-4da3-9f5b-1adfc9851694-manager-config\") pod \"lws-controller-manager-7bc87557cd-pc7zb\" (UID: \"6446a118-60ed-4da3-9f5b-1adfc9851694\") " pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:17.291781 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.291759    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6446a118-60ed-4da3-9f5b-1adfc9851694-metrics-cert\") pod \"lws-controller-manager-7bc87557cd-pc7zb\" (UID: \"6446a118-60ed-4da3-9f5b-1adfc9851694\") " pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:17.291940 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.291920    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6446a118-60ed-4da3-9f5b-1adfc9851694-cert\") pod \"lws-controller-manager-7bc87557cd-pc7zb\" (UID: \"6446a118-60ed-4da3-9f5b-1adfc9851694\") " pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:17.298246 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.298218    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjzzv\" (UniqueName: \"kubernetes.io/projected/6446a118-60ed-4da3-9f5b-1adfc9851694-kube-api-access-hjzzv\") pod \"lws-controller-manager-7bc87557cd-pc7zb\" (UID: \"6446a118-60ed-4da3-9f5b-1adfc9851694\") " pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:17.390228 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.390144    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:17.680519 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:17:17.680482    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6446a118_60ed_4da3_9f5b_1adfc9851694.slice/crio-81d0efd2f67e747e879bd61331d6b2b237ce1ead5e83ee63f6edb7ef03eb1c41 WatchSource:0}: Error finding container 81d0efd2f67e747e879bd61331d6b2b237ce1ead5e83ee63f6edb7ef03eb1c41: Status 404 returned error can't find the container with id 81d0efd2f67e747e879bd61331d6b2b237ce1ead5e83ee63f6edb7ef03eb1c41
Sep 09 16:17:17.682095 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:17.682071    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"]
Sep 09 16:17:18.147215 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:18.147177    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb" event={"ID":"6446a118-60ed-4da3-9f5b-1adfc9851694","Type":"ContainerStarted","Data":"81d0efd2f67e747e879bd61331d6b2b237ce1ead5e83ee63f6edb7ef03eb1c41"}
Sep 09 16:17:21.159026 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:21.158942    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb" event={"ID":"6446a118-60ed-4da3-9f5b-1adfc9851694","Type":"ContainerStarted","Data":"356bf737d1eabb27e2b9883fbc0146b5fc2722fd6b2c95982e065b4f2005b6cb"}
Sep 09 16:17:21.159372 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:21.159075    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:21.182994 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:21.182941    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb" podStartSLOduration=1.109019916 podStartE2EDuration="4.182927289s" totalImagesPullingTime="3.073907373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:17:21.181889391 +0000 UTC m=+517.117507841" watchObservedRunningTime="2026-09-09 16:17:21.182927289 +0000 UTC m=+517.118545705"
Sep 09 16:17:24.150765 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.150731    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"]
Sep 09 16:17:24.153869 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.153853    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.163562 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.163521    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 16:17:24.163673 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.163567    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vdnpc\""
Sep 09 16:17:24.163673 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.163613    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 16:17:24.163673 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.163648    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 16:17:24.163932 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.163917    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 16:17:24.195766 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.195739    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"]
Sep 09 16:17:24.244768 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.244742    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17c62f0e-b8f0-4669-bce9-c37adcd58a36-webhook-cert\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.244871 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.244801    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17c62f0e-b8f0-4669-bce9-c37adcd58a36-manifests\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.244914 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.244876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17c62f0e-b8f0-4669-bce9-c37adcd58a36-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.244914 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.244908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxz7k\" (UniqueName: \"kubernetes.io/projected/17c62f0e-b8f0-4669-bce9-c37adcd58a36-kube-api-access-lxz7k\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.244994 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.244931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17c62f0e-b8f0-4669-bce9-c37adcd58a36-tmp\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.345815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.345786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17c62f0e-b8f0-4669-bce9-c37adcd58a36-webhook-cert\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.345951 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.345825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17c62f0e-b8f0-4669-bce9-c37adcd58a36-manifests\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.345951 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.345849    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17c62f0e-b8f0-4669-bce9-c37adcd58a36-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.345951 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.345869    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxz7k\" (UniqueName: \"kubernetes.io/projected/17c62f0e-b8f0-4669-bce9-c37adcd58a36-kube-api-access-lxz7k\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.345951 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.345929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17c62f0e-b8f0-4669-bce9-c37adcd58a36-tmp\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.346242 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.346219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17c62f0e-b8f0-4669-bce9-c37adcd58a36-manifests\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.348232 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.348208    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17c62f0e-b8f0-4669-bce9-c37adcd58a36-tmp\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.348332 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.348298    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17c62f0e-b8f0-4669-bce9-c37adcd58a36-webhook-cert\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.348497 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.348479    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17c62f0e-b8f0-4669-bce9-c37adcd58a36-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.363179 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.363157    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxz7k\" (UniqueName: \"kubernetes.io/projected/17c62f0e-b8f0-4669-bce9-c37adcd58a36-kube-api-access-lxz7k\") pod \"opendatahub-operator-controller-manager-67c7bcf47f-dflt7\" (UID: \"17c62f0e-b8f0-4669-bce9-c37adcd58a36\") " pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.463410 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.463341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:24.777322 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:24.777296    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"]
Sep 09 16:17:24.779736 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:17:24.779711    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c62f0e_b8f0_4669_bce9_c37adcd58a36.slice/crio-9fd9cac446c9716846b804df465064c752cc40a9559ce144d3e8f9f9407eedff WatchSource:0}: Error finding container 9fd9cac446c9716846b804df465064c752cc40a9559ce144d3e8f9f9407eedff: Status 404 returned error can't find the container with id 9fd9cac446c9716846b804df465064c752cc40a9559ce144d3e8f9f9407eedff
Sep 09 16:17:25.177775 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:25.177670    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7" event={"ID":"17c62f0e-b8f0-4669-bce9-c37adcd58a36","Type":"ContainerStarted","Data":"9fd9cac446c9716846b804df465064c752cc40a9559ce144d3e8f9f9407eedff"}
Sep 09 16:17:28.189585 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:28.189497    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7" event={"ID":"17c62f0e-b8f0-4669-bce9-c37adcd58a36","Type":"ContainerStarted","Data":"8778338b03a01b1db0d41d22d994b938315ad36f85bd56ab81b250108154fab2"}
Sep 09 16:17:29.194273 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:29.194177    2576 generic.go:356] "Generic (PLEG): container finished" podID="17c62f0e-b8f0-4669-bce9-c37adcd58a36" containerID="8778338b03a01b1db0d41d22d994b938315ad36f85bd56ab81b250108154fab2" exitCode=0
Sep 09 16:17:29.194623 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:29.194277    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7" event={"ID":"17c62f0e-b8f0-4669-bce9-c37adcd58a36","Type":"ContainerDied","Data":"8778338b03a01b1db0d41d22d994b938315ad36f85bd56ab81b250108154fab2"}
Sep 09 16:17:30.200268 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:30.200231    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7" event={"ID":"17c62f0e-b8f0-4669-bce9-c37adcd58a36","Type":"ContainerStarted","Data":"f4e821a8dac394d1ea63bfe815f043ee3c0d9996a3ee336b9674aa301332fd63"}
Sep 09 16:17:30.200642 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:30.200347    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:30.222911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:30.222861    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7" podStartSLOduration=1.895405475 podStartE2EDuration="6.222845432s" totalImagesPullingTime="3.288443473s" totalInitContainerRuntime="1.038996484s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:17:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:17:30.221467223 +0000 UTC m=+526.157085649" watchObservedRunningTime="2026-09-09 16:17:30.222845432 +0000 UTC m=+526.158463917"
Sep 09 16:17:32.165379 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:32.165351    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7bc87557cd-pc7zb"
Sep 09 16:17:41.205054 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:41.204979    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67c7bcf47f-dflt7"
Sep 09 16:17:44.070142 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.070105    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n"]
Sep 09 16:17:44.154835 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.154797    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n"]
Sep 09 16:17:44.154997 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.154898    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n"
Sep 09 16:17:44.157858 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.157775    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 16:17:44.157858 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.157795    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 16:17:44.157858 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.157808    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 16:17:44.158191 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.158174    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 16:17:44.158849 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.158830    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-x7q7q\""
Sep 09 16:17:44.290796 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.290762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4w6b\" (UniqueName: \"kubernetes.io/projected/b2893fe7-94e2-4115-8362-925f3908b0a6-kube-api-access-c4w6b\") pod \"kube-auth-proxy-54fc5d6dc5-4qr8n\" (UID: \"b2893fe7-94e2-4115-8362-925f3908b0a6\") " pod="openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n"
Sep 09 16:17:44.290965 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.290829    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2893fe7-94e2-4115-8362-925f3908b0a6-tls-certs\") pod \"kube-auth-proxy-54fc5d6dc5-4qr8n\" (UID: \"b2893fe7-94e2-4115-8362-925f3908b0a6\") " pod="openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n"
Sep 09 16:17:44.290965 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.290863    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2893fe7-94e2-4115-8362-925f3908b0a6-tmp\") pod \"kube-auth-proxy-54fc5d6dc5-4qr8n\" (UID: \"b2893fe7-94e2-4115-8362-925f3908b0a6\") " pod="openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n"
Sep 09 16:17:44.391578 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.391492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4w6b\" (UniqueName: \"kubernetes.io/projected/b2893fe7-94e2-4115-8362-925f3908b0a6-kube-api-access-c4w6b\") pod \"kube-auth-proxy-54fc5d6dc5-4qr8n\" (UID: \"b2893fe7-94e2-4115-8362-925f3908b0a6\") " pod="openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n"
Sep 09 16:17:44.391578 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.391543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2893fe7-94e2-4115-8362-925f3908b0a6-tls-certs\") pod \"kube-auth-proxy-54fc5d6dc5-4qr8n\" (UID: \"b2893fe7-94e2-4115-8362-925f3908b0a6\") " pod="openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n"
Sep 09 16:17:44.391578 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.391561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2893fe7-94e2-4115-8362-925f3908b0a6-tmp\") pod \"kube-auth-proxy-54fc5d6dc5-4qr8n\" (UID: \"b2893fe7-94e2-4115-8362-925f3908b0a6\") " pod="openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n"
Sep 09 16:17:44.394069 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.394041    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2893fe7-94e2-4115-8362-925f3908b0a6-tmp\") pod \"kube-auth-proxy-54fc5d6dc5-4qr8n\" (UID: \"b2893fe7-94e2-4115-8362-925f3908b0a6\") " pod="openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n"
Sep 09 16:17:44.394183 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.394171    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2893fe7-94e2-4115-8362-925f3908b0a6-tls-certs\") pod \"kube-auth-proxy-54fc5d6dc5-4qr8n\" (UID: \"b2893fe7-94e2-4115-8362-925f3908b0a6\") " pod="openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n"
Sep 09 16:17:44.400234 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.400210    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4w6b\" (UniqueName: \"kubernetes.io/projected/b2893fe7-94e2-4115-8362-925f3908b0a6-kube-api-access-c4w6b\") pod \"kube-auth-proxy-54fc5d6dc5-4qr8n\" (UID: \"b2893fe7-94e2-4115-8362-925f3908b0a6\") " pod="openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n"
Sep 09 16:17:44.468165 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.468142    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-x7q7q\""
Sep 09 16:17:44.476072 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.476052    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n"
Sep 09 16:17:44.776760 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:44.776728    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n"]
Sep 09 16:17:44.779512 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:17:44.779483    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2893fe7_94e2_4115_8362_925f3908b0a6.slice/crio-99140902e874a97b729354b648a5c52b5d9a2deb1a54dda74b70cb4948fbb237 WatchSource:0}: Error finding container 99140902e874a97b729354b648a5c52b5d9a2deb1a54dda74b70cb4948fbb237: Status 404 returned error can't find the container with id 99140902e874a97b729354b648a5c52b5d9a2deb1a54dda74b70cb4948fbb237
Sep 09 16:17:45.250298 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:45.250262    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n" event={"ID":"b2893fe7-94e2-4115-8362-925f3908b0a6","Type":"ContainerStarted","Data":"99140902e874a97b729354b648a5c52b5d9a2deb1a54dda74b70cb4948fbb237"}
Sep 09 16:17:47.882826 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:47.882803    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 16:17:48.263320 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:48.263287    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n" event={"ID":"b2893fe7-94e2-4115-8362-925f3908b0a6","Type":"ContainerStarted","Data":"81bd1e5dc004753a89017b41f3f80d5f9a6491da8d51e8277ca272e2b7f4de15"}
Sep 09 16:17:48.279900 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:17:48.279857    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54fc5d6dc5-4qr8n" podStartSLOduration=1.181371395 podStartE2EDuration="4.279842982s" totalImagesPullingTime="3.098471587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:17:48.278924196 +0000 UTC m=+544.214542623" watchObservedRunningTime="2026-09-09 16:17:48.279842982 +0000 UTC m=+544.215461398"
Sep 09 16:19:24.911311 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:24.911277    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx"]
Sep 09 16:19:24.914376 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:24.914358    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx"
Sep 09 16:19:24.916550 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:24.916522    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 16:19:24.916550 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:24.916551    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bz25f\""
Sep 09 16:19:24.916735 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:24.916555    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 16:19:24.917440 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:24.917424    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 16:19:24.917482 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:24.917451    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 16:19:24.921908 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:24.921887    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx"]
Sep 09 16:19:25.029442 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:25.029410    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp29v\" (UniqueName: \"kubernetes.io/projected/5afd1b8d-a975-4ea0-84da-d54d43a5e78b-kube-api-access-rp29v\") pod \"kuadrant-console-plugin-fbdd4f445-rz6hx\" (UID: \"5afd1b8d-a975-4ea0-84da-d54d43a5e78b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx"
Sep 09 16:19:25.029607 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:25.029447    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5afd1b8d-a975-4ea0-84da-d54d43a5e78b-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rz6hx\" (UID: \"5afd1b8d-a975-4ea0-84da-d54d43a5e78b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx"
Sep 09 16:19:25.029607 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:25.029473    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5afd1b8d-a975-4ea0-84da-d54d43a5e78b-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-rz6hx\" (UID: \"5afd1b8d-a975-4ea0-84da-d54d43a5e78b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx"
Sep 09 16:19:25.130370 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:25.130329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp29v\" (UniqueName: \"kubernetes.io/projected/5afd1b8d-a975-4ea0-84da-d54d43a5e78b-kube-api-access-rp29v\") pod \"kuadrant-console-plugin-fbdd4f445-rz6hx\" (UID: \"5afd1b8d-a975-4ea0-84da-d54d43a5e78b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx"
Sep 09 16:19:25.130532 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:25.130379    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5afd1b8d-a975-4ea0-84da-d54d43a5e78b-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rz6hx\" (UID: \"5afd1b8d-a975-4ea0-84da-d54d43a5e78b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx"
Sep 09 16:19:25.130532 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:25.130408    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5afd1b8d-a975-4ea0-84da-d54d43a5e78b-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-rz6hx\" (UID: \"5afd1b8d-a975-4ea0-84da-d54d43a5e78b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx"
Sep 09 16:19:25.131088 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:25.131068    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5afd1b8d-a975-4ea0-84da-d54d43a5e78b-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-rz6hx\" (UID: \"5afd1b8d-a975-4ea0-84da-d54d43a5e78b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx"
Sep 09 16:19:25.132917 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:25.132894    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5afd1b8d-a975-4ea0-84da-d54d43a5e78b-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rz6hx\" (UID: \"5afd1b8d-a975-4ea0-84da-d54d43a5e78b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx"
Sep 09 16:19:25.138265 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:25.138246    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp29v\" (UniqueName: \"kubernetes.io/projected/5afd1b8d-a975-4ea0-84da-d54d43a5e78b-kube-api-access-rp29v\") pod \"kuadrant-console-plugin-fbdd4f445-rz6hx\" (UID: \"5afd1b8d-a975-4ea0-84da-d54d43a5e78b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx"
Sep 09 16:19:25.223238 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:25.223214    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx"
Sep 09 16:19:25.489829 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:25.489752    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx"]
Sep 09 16:19:25.493505 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:19:25.493473    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5afd1b8d_a975_4ea0_84da_d54d43a5e78b.slice/crio-699e35c2910ebafc6e67a5cc5195d1a62d7b0d9fe8962a059b9a14d3dcc58e4b WatchSource:0}: Error finding container 699e35c2910ebafc6e67a5cc5195d1a62d7b0d9fe8962a059b9a14d3dcc58e4b: Status 404 returned error can't find the container with id 699e35c2910ebafc6e67a5cc5195d1a62d7b0d9fe8962a059b9a14d3dcc58e4b
Sep 09 16:19:25.598993 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:25.598965    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx" event={"ID":"5afd1b8d-a975-4ea0-84da-d54d43a5e78b","Type":"ContainerStarted","Data":"699e35c2910ebafc6e67a5cc5195d1a62d7b0d9fe8962a059b9a14d3dcc58e4b"}
Sep 09 16:19:30.620573 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:30.620537    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx" event={"ID":"5afd1b8d-a975-4ea0-84da-d54d43a5e78b","Type":"ContainerStarted","Data":"1142c667c00543791bbb388bcffb796bda642cef1c7d1ef8121ca0aee7d7e724"}
Sep 09 16:19:30.638056 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:19:30.638014    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rz6hx" podStartSLOduration=1.728231884 podStartE2EDuration="6.63799965s" totalImagesPullingTime="4.909767766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:30.636915935 +0000 UTC m=+646.572534353" watchObservedRunningTime="2026-09-09 16:19:30.63799965 +0000 UTC m=+646.573618066"
Sep 09 16:20:22.551992 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.551961    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wqcf6"]
Sep 09 16:20:22.553883 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.553867    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-wqcf6"
Sep 09 16:20:22.556126 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.556106    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nftrl\""
Sep 09 16:20:22.562083 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.562055    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wqcf6"]
Sep 09 16:20:22.648065 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.648032    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9wmn\" (UniqueName: \"kubernetes.io/projected/86099b9d-bc22-42bd-87b2-7c5c4fa506d4-kube-api-access-f9wmn\") pod \"authorino-7bcfd8f9cd-wqcf6\" (UID: \"86099b9d-bc22-42bd-87b2-7c5c4fa506d4\") " pod="kuadrant-system/authorino-7bcfd8f9cd-wqcf6"
Sep 09 16:20:22.672059 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.672031    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-wgts9"]
Sep 09 16:20:22.674138 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.674118    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wgts9"
Sep 09 16:20:22.682183 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.682157    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wgts9"]
Sep 09 16:20:22.748789 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.748757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9wmn\" (UniqueName: \"kubernetes.io/projected/86099b9d-bc22-42bd-87b2-7c5c4fa506d4-kube-api-access-f9wmn\") pod \"authorino-7bcfd8f9cd-wqcf6\" (UID: \"86099b9d-bc22-42bd-87b2-7c5c4fa506d4\") " pod="kuadrant-system/authorino-7bcfd8f9cd-wqcf6"
Sep 09 16:20:22.756766 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.756741    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9wmn\" (UniqueName: \"kubernetes.io/projected/86099b9d-bc22-42bd-87b2-7c5c4fa506d4-kube-api-access-f9wmn\") pod \"authorino-7bcfd8f9cd-wqcf6\" (UID: \"86099b9d-bc22-42bd-87b2-7c5c4fa506d4\") " pod="kuadrant-system/authorino-7bcfd8f9cd-wqcf6"
Sep 09 16:20:22.849467 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.849376    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxk8d\" (UniqueName: \"kubernetes.io/projected/e785d84f-f078-4919-876d-7a60f05035d5-kube-api-access-sxk8d\") pod \"authorino-59c974bd48-wgts9\" (UID: \"e785d84f-f078-4919-876d-7a60f05035d5\") " pod="kuadrant-system/authorino-59c974bd48-wgts9"
Sep 09 16:20:22.863313 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.863287    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-wqcf6"
Sep 09 16:20:22.950078 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.950047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxk8d\" (UniqueName: \"kubernetes.io/projected/e785d84f-f078-4919-876d-7a60f05035d5-kube-api-access-sxk8d\") pod \"authorino-59c974bd48-wgts9\" (UID: \"e785d84f-f078-4919-876d-7a60f05035d5\") " pod="kuadrant-system/authorino-59c974bd48-wgts9"
Sep 09 16:20:22.957768 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.957742    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxk8d\" (UniqueName: \"kubernetes.io/projected/e785d84f-f078-4919-876d-7a60f05035d5-kube-api-access-sxk8d\") pod \"authorino-59c974bd48-wgts9\" (UID: \"e785d84f-f078-4919-876d-7a60f05035d5\") " pod="kuadrant-system/authorino-59c974bd48-wgts9"
Sep 09 16:20:22.983165 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:22.983141    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wgts9"
Sep 09 16:20:23.132403 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:23.132335    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wqcf6"]
Sep 09 16:20:23.135062 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:20:23.135033    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86099b9d_bc22_42bd_87b2_7c5c4fa506d4.slice/crio-a7f2b65cb7206df5ddf0c4b4bcc2843b0ca5ea6b3c5116af2710dcfe1197c43e WatchSource:0}: Error finding container a7f2b65cb7206df5ddf0c4b4bcc2843b0ca5ea6b3c5116af2710dcfe1197c43e: Status 404 returned error can't find the container with id a7f2b65cb7206df5ddf0c4b4bcc2843b0ca5ea6b3c5116af2710dcfe1197c43e
Sep 09 16:20:23.136408 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:23.136389    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:20:23.251844 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:23.250060    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wgts9"]
Sep 09 16:20:23.255278 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:20:23.255255    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode785d84f_f078_4919_876d_7a60f05035d5.slice/crio-7ed97a891c838ea6adb2d56736d5b9423748282c426d8dd9afa6d0c09e9424a7 WatchSource:0}: Error finding container 7ed97a891c838ea6adb2d56736d5b9423748282c426d8dd9afa6d0c09e9424a7: Status 404 returned error can't find the container with id 7ed97a891c838ea6adb2d56736d5b9423748282c426d8dd9afa6d0c09e9424a7
Sep 09 16:20:23.807006 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:23.806968    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wgts9" event={"ID":"e785d84f-f078-4919-876d-7a60f05035d5","Type":"ContainerStarted","Data":"7ed97a891c838ea6adb2d56736d5b9423748282c426d8dd9afa6d0c09e9424a7"}
Sep 09 16:20:23.808299 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:23.808272    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-wqcf6" event={"ID":"86099b9d-bc22-42bd-87b2-7c5c4fa506d4","Type":"ContainerStarted","Data":"a7f2b65cb7206df5ddf0c4b4bcc2843b0ca5ea6b3c5116af2710dcfe1197c43e"}
Sep 09 16:20:26.822172 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:26.822134    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wgts9" event={"ID":"e785d84f-f078-4919-876d-7a60f05035d5","Type":"ContainerStarted","Data":"488f0ffccadf8aa3bb9b8697a20676a36d4fd53064d4f20d426d55ba99cbb110"}
Sep 09 16:20:26.823588 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:26.823561    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-wqcf6" event={"ID":"86099b9d-bc22-42bd-87b2-7c5c4fa506d4","Type":"ContainerStarted","Data":"649dfabe53fcfd3f0a1cf86c9f22b05b17ba8c95ec251586164143c37d7b3474"}
Sep 09 16:20:26.836718 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:26.836656    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-wgts9" podStartSLOduration=2.268580836 podStartE2EDuration="4.836642625s" totalImagesPullingTime="2.568061789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:20:26.836256576 +0000 UTC m=+702.771875005" watchObservedRunningTime="2026-09-09 16:20:26.836642625 +0000 UTC m=+702.772261040"
Sep 09 16:20:26.851651 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:26.851609    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-wqcf6" podStartSLOduration=2.154008004 podStartE2EDuration="4.851591875s" totalImagesPullingTime="2.697583871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:20:26.850223638 +0000 UTC m=+702.785842055" watchObservedRunningTime="2026-09-09 16:20:26.851591875 +0000 UTC m=+702.787210292"
Sep 09 16:20:26.874240 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:26.874215    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wqcf6"]
Sep 09 16:20:28.831392 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:28.831349    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-wqcf6" podUID="86099b9d-bc22-42bd-87b2-7c5c4fa506d4" containerName="authorino" containerID="cri-o://649dfabe53fcfd3f0a1cf86c9f22b05b17ba8c95ec251586164143c37d7b3474" gracePeriod=30
Sep 09 16:20:29.071656 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:29.071632    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-wqcf6"
Sep 09 16:20:29.197762 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:29.197667    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9wmn\" (UniqueName: \"kubernetes.io/projected/86099b9d-bc22-42bd-87b2-7c5c4fa506d4-kube-api-access-f9wmn\") pod \"86099b9d-bc22-42bd-87b2-7c5c4fa506d4\" (UID: \"86099b9d-bc22-42bd-87b2-7c5c4fa506d4\") "
Sep 09 16:20:29.199960 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:29.199933    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86099b9d-bc22-42bd-87b2-7c5c4fa506d4-kube-api-access-f9wmn" (OuterVolumeSpecName: "kube-api-access-f9wmn") pod "86099b9d-bc22-42bd-87b2-7c5c4fa506d4" (UID: "86099b9d-bc22-42bd-87b2-7c5c4fa506d4"). InnerVolumeSpecName "kube-api-access-f9wmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:20:29.298666 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:29.298631    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9wmn\" (UniqueName: \"kubernetes.io/projected/86099b9d-bc22-42bd-87b2-7c5c4fa506d4-kube-api-access-f9wmn\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:20:29.835798 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:29.835762    2576 generic.go:356] "Generic (PLEG): container finished" podID="86099b9d-bc22-42bd-87b2-7c5c4fa506d4" containerID="649dfabe53fcfd3f0a1cf86c9f22b05b17ba8c95ec251586164143c37d7b3474" exitCode=0
Sep 09 16:20:29.836210 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:29.835803    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-wqcf6"
Sep 09 16:20:29.836210 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:29.835850    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-wqcf6" event={"ID":"86099b9d-bc22-42bd-87b2-7c5c4fa506d4","Type":"ContainerDied","Data":"649dfabe53fcfd3f0a1cf86c9f22b05b17ba8c95ec251586164143c37d7b3474"}
Sep 09 16:20:29.836210 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:29.835884    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-wqcf6" event={"ID":"86099b9d-bc22-42bd-87b2-7c5c4fa506d4","Type":"ContainerDied","Data":"a7f2b65cb7206df5ddf0c4b4bcc2843b0ca5ea6b3c5116af2710dcfe1197c43e"}
Sep 09 16:20:29.836210 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:29.835901    2576 scope.go:117] "RemoveContainer" containerID="649dfabe53fcfd3f0a1cf86c9f22b05b17ba8c95ec251586164143c37d7b3474"
Sep 09 16:20:29.844599 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:29.844579    2576 scope.go:117] "RemoveContainer" containerID="649dfabe53fcfd3f0a1cf86c9f22b05b17ba8c95ec251586164143c37d7b3474"
Sep 09 16:20:29.844896 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:20:29.844879    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649dfabe53fcfd3f0a1cf86c9f22b05b17ba8c95ec251586164143c37d7b3474\": container with ID starting with 649dfabe53fcfd3f0a1cf86c9f22b05b17ba8c95ec251586164143c37d7b3474 not found: ID does not exist" containerID="649dfabe53fcfd3f0a1cf86c9f22b05b17ba8c95ec251586164143c37d7b3474"
Sep 09 16:20:29.844947 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:29.844903    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649dfabe53fcfd3f0a1cf86c9f22b05b17ba8c95ec251586164143c37d7b3474"} err="failed to get container status \"649dfabe53fcfd3f0a1cf86c9f22b05b17ba8c95ec251586164143c37d7b3474\": rpc error: code = NotFound desc = could not find container \"649dfabe53fcfd3f0a1cf86c9f22b05b17ba8c95ec251586164143c37d7b3474\": container with ID starting with 649dfabe53fcfd3f0a1cf86c9f22b05b17ba8c95ec251586164143c37d7b3474 not found: ID does not exist"
Sep 09 16:20:29.855643 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:29.855618    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wqcf6"]
Sep 09 16:20:29.859264 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:29.859242    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wqcf6"]
Sep 09 16:20:30.552153 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:30.552119    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86099b9d-bc22-42bd-87b2-7c5c4fa506d4" path="/var/lib/kubelet/pods/86099b9d-bc22-42bd-87b2-7c5c4fa506d4/volumes"
Sep 09 16:20:58.805113 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.805067    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-ndcrb"]
Sep 09 16:20:58.805611 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.805325    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86099b9d-bc22-42bd-87b2-7c5c4fa506d4" containerName="authorino"
Sep 09 16:20:58.805611 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.805335    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="86099b9d-bc22-42bd-87b2-7c5c4fa506d4" containerName="authorino"
Sep 09 16:20:58.805611 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.805379    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="86099b9d-bc22-42bd-87b2-7c5c4fa506d4" containerName="authorino"
Sep 09 16:20:58.812755 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.812732    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ndcrb"
Sep 09 16:20:58.815255 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.815228    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-ndcrb"]
Sep 09 16:20:58.853559 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.853528    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-ndcrb"]
Sep 09 16:20:58.853799 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:20:58.853756    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-bgh6d], unattached volumes=[], failed to process volumes=[kube-api-access-bgh6d]: context canceled" pod="kuadrant-system/authorino-557f876f69-ndcrb" podUID="bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5"
Sep 09 16:20:58.878345 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.878308    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5d6qd"]
Sep 09 16:20:58.881667 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.881648    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-5d6qd"
Sep 09 16:20:58.883901 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.883880    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 16:20:58.889205 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.889186    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5d6qd"]
Sep 09 16:20:58.910640 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.910595    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgh6d\" (UniqueName: \"kubernetes.io/projected/bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5-kube-api-access-bgh6d\") pod \"authorino-557f876f69-ndcrb\" (UID: \"bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5\") " pod="kuadrant-system/authorino-557f876f69-ndcrb"
Sep 09 16:20:58.910796 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.910730    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdrkc\" (UniqueName: \"kubernetes.io/projected/1e496b37-dbaf-4b8f-9bdc-4d049c4cf178-kube-api-access-vdrkc\") pod \"authorino-6d4dfc649f-5d6qd\" (UID: \"1e496b37-dbaf-4b8f-9bdc-4d049c4cf178\") " pod="kuadrant-system/authorino-6d4dfc649f-5d6qd"
Sep 09 16:20:58.910796 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.910764    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e496b37-dbaf-4b8f-9bdc-4d049c4cf178-tls-cert\") pod \"authorino-6d4dfc649f-5d6qd\" (UID: \"1e496b37-dbaf-4b8f-9bdc-4d049c4cf178\") " pod="kuadrant-system/authorino-6d4dfc649f-5d6qd"
Sep 09 16:20:58.938447 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.938412    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ndcrb"
Sep 09 16:20:58.947138 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:58.947117    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ndcrb"
Sep 09 16:20:59.011288 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.011258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdrkc\" (UniqueName: \"kubernetes.io/projected/1e496b37-dbaf-4b8f-9bdc-4d049c4cf178-kube-api-access-vdrkc\") pod \"authorino-6d4dfc649f-5d6qd\" (UID: \"1e496b37-dbaf-4b8f-9bdc-4d049c4cf178\") " pod="kuadrant-system/authorino-6d4dfc649f-5d6qd"
Sep 09 16:20:59.011472 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.011296    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e496b37-dbaf-4b8f-9bdc-4d049c4cf178-tls-cert\") pod \"authorino-6d4dfc649f-5d6qd\" (UID: \"1e496b37-dbaf-4b8f-9bdc-4d049c4cf178\") " pod="kuadrant-system/authorino-6d4dfc649f-5d6qd"
Sep 09 16:20:59.011472 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.011321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgh6d\" (UniqueName: \"kubernetes.io/projected/bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5-kube-api-access-bgh6d\") pod \"authorino-557f876f69-ndcrb\" (UID: \"bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5\") " pod="kuadrant-system/authorino-557f876f69-ndcrb"
Sep 09 16:20:59.013907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.013886    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e496b37-dbaf-4b8f-9bdc-4d049c4cf178-tls-cert\") pod \"authorino-6d4dfc649f-5d6qd\" (UID: \"1e496b37-dbaf-4b8f-9bdc-4d049c4cf178\") " pod="kuadrant-system/authorino-6d4dfc649f-5d6qd"
Sep 09 16:20:59.019646 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.019625    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdrkc\" (UniqueName: \"kubernetes.io/projected/1e496b37-dbaf-4b8f-9bdc-4d049c4cf178-kube-api-access-vdrkc\") pod \"authorino-6d4dfc649f-5d6qd\" (UID: \"1e496b37-dbaf-4b8f-9bdc-4d049c4cf178\") " pod="kuadrant-system/authorino-6d4dfc649f-5d6qd"
Sep 09 16:20:59.019921 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.019904    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgh6d\" (UniqueName: \"kubernetes.io/projected/bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5-kube-api-access-bgh6d\") pod \"authorino-557f876f69-ndcrb\" (UID: \"bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5\") " pod="kuadrant-system/authorino-557f876f69-ndcrb"
Sep 09 16:20:59.076000 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.075922    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5d6qd"]
Sep 09 16:20:59.076158 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.076147    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-5d6qd"
Sep 09 16:20:59.103785 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.103756    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7576868b45-r74x2"]
Sep 09 16:20:59.107185 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.107170    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7576868b45-r74x2"
Sep 09 16:20:59.111909 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.111875    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgh6d\" (UniqueName: \"kubernetes.io/projected/bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5-kube-api-access-bgh6d\") pod \"bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5\" (UID: \"bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5\") "
Sep 09 16:20:59.113224 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.113198    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7576868b45-r74x2"]
Sep 09 16:20:59.114625 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.114460    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5-kube-api-access-bgh6d" (OuterVolumeSpecName: "kube-api-access-bgh6d") pod "bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5" (UID: "bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5"). InnerVolumeSpecName "kube-api-access-bgh6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:20:59.213372 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.213337    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkcl\" (UniqueName: \"kubernetes.io/projected/35659680-4927-4e1d-b21a-479365e0f2cf-kube-api-access-lhkcl\") pod \"authorino-7576868b45-r74x2\" (UID: \"35659680-4927-4e1d-b21a-479365e0f2cf\") " pod="kuadrant-system/authorino-7576868b45-r74x2"
Sep 09 16:20:59.213372 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.213377    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35659680-4927-4e1d-b21a-479365e0f2cf-tls-cert\") pod \"authorino-7576868b45-r74x2\" (UID: \"35659680-4927-4e1d-b21a-479365e0f2cf\") " pod="kuadrant-system/authorino-7576868b45-r74x2"
Sep 09 16:20:59.213581 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.213406    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgh6d\" (UniqueName: \"kubernetes.io/projected/bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5-kube-api-access-bgh6d\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:20:59.314166 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.314128    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkcl\" (UniqueName: \"kubernetes.io/projected/35659680-4927-4e1d-b21a-479365e0f2cf-kube-api-access-lhkcl\") pod \"authorino-7576868b45-r74x2\" (UID: \"35659680-4927-4e1d-b21a-479365e0f2cf\") " pod="kuadrant-system/authorino-7576868b45-r74x2"
Sep 09 16:20:59.314166 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.314164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35659680-4927-4e1d-b21a-479365e0f2cf-tls-cert\") pod \"authorino-7576868b45-r74x2\" (UID: \"35659680-4927-4e1d-b21a-479365e0f2cf\") " pod="kuadrant-system/authorino-7576868b45-r74x2"
Sep 09 16:20:59.316629 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.316606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35659680-4927-4e1d-b21a-479365e0f2cf-tls-cert\") pod \"authorino-7576868b45-r74x2\" (UID: \"35659680-4927-4e1d-b21a-479365e0f2cf\") " pod="kuadrant-system/authorino-7576868b45-r74x2"
Sep 09 16:20:59.322513 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.322486    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkcl\" (UniqueName: \"kubernetes.io/projected/35659680-4927-4e1d-b21a-479365e0f2cf-kube-api-access-lhkcl\") pod \"authorino-7576868b45-r74x2\" (UID: \"35659680-4927-4e1d-b21a-479365e0f2cf\") " pod="kuadrant-system/authorino-7576868b45-r74x2"
Sep 09 16:20:59.349618 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.349559    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5d6qd"]
Sep 09 16:20:59.352354 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:20:59.352329    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e496b37_dbaf_4b8f_9bdc_4d049c4cf178.slice/crio-8aa3c40cf4f17d0bf86bdc4db00193512b5fa77c2706b03ee8818d75216458cf WatchSource:0}: Error finding container 8aa3c40cf4f17d0bf86bdc4db00193512b5fa77c2706b03ee8818d75216458cf: Status 404 returned error can't find the container with id 8aa3c40cf4f17d0bf86bdc4db00193512b5fa77c2706b03ee8818d75216458cf
Sep 09 16:20:59.422677 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.422652    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7576868b45-r74x2"
Sep 09 16:20:59.690672 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.690498    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7576868b45-r74x2"]
Sep 09 16:20:59.693100 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:20:59.693064    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35659680_4927_4e1d_b21a_479365e0f2cf.slice/crio-dc871999f841c22bea95df1f17c3108f55cc7bf9629b38eb7bb1f12f854853ad WatchSource:0}: Error finding container dc871999f841c22bea95df1f17c3108f55cc7bf9629b38eb7bb1f12f854853ad: Status 404 returned error can't find the container with id dc871999f841c22bea95df1f17c3108f55cc7bf9629b38eb7bb1f12f854853ad
Sep 09 16:20:59.943088 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.943027    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7576868b45-r74x2" event={"ID":"35659680-4927-4e1d-b21a-479365e0f2cf","Type":"ContainerStarted","Data":"dc871999f841c22bea95df1f17c3108f55cc7bf9629b38eb7bb1f12f854853ad"}
Sep 09 16:20:59.944242 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.944217    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-5d6qd" event={"ID":"1e496b37-dbaf-4b8f-9bdc-4d049c4cf178","Type":"ContainerStarted","Data":"8aa3c40cf4f17d0bf86bdc4db00193512b5fa77c2706b03ee8818d75216458cf"}
Sep 09 16:20:59.944346 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.944231    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ndcrb"
Sep 09 16:20:59.971585 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.971561    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-ndcrb"]
Sep 09 16:20:59.975812 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:20:59.975784    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-ndcrb"]
Sep 09 16:21:00.552584 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:00.552546    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5" path="/var/lib/kubelet/pods/bdc9cb55-dc24-4ea7-a92e-89b1a2d3abb5/volumes"
Sep 09 16:21:00.949867 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:00.949772    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7576868b45-r74x2" event={"ID":"35659680-4927-4e1d-b21a-479365e0f2cf","Type":"ContainerStarted","Data":"d75c84b0a930334d7294f21786bc18dcf1601efc5d0346897fbabb2c7b92371e"}
Sep 09 16:21:00.951195 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:00.951171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-5d6qd" event={"ID":"1e496b37-dbaf-4b8f-9bdc-4d049c4cf178","Type":"ContainerStarted","Data":"b2378cf0910b5d212cd0a62c7cd52c88fb88dc606f64be9aa13890a9ebba6ace"}
Sep 09 16:21:00.951299 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:00.951251    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d4dfc649f-5d6qd" podUID="1e496b37-dbaf-4b8f-9bdc-4d049c4cf178" containerName="authorino" containerID="cri-o://b2378cf0910b5d212cd0a62c7cd52c88fb88dc606f64be9aa13890a9ebba6ace" gracePeriod=30
Sep 09 16:21:00.967137 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:00.967097    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7576868b45-r74x2" podStartSLOduration=1.4918781540000001 podStartE2EDuration="1.967081878s" totalImagesPullingTime="475.203724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:21:00.965109947 +0000 UTC m=+736.900728342" watchObservedRunningTime="2026-09-09 16:21:00.967081878 +0000 UTC m=+736.902700294"
Sep 09 16:21:00.983545 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:00.983491    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d4dfc649f-5d6qd" podStartSLOduration=2.432621185 podStartE2EDuration="2.983474374s" totalImagesPullingTime="550.853189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:21:00.980919059 +0000 UTC m=+736.916537489" watchObservedRunningTime="2026-09-09 16:21:00.983474374 +0000 UTC m=+736.919092791"
Sep 09 16:21:00.993157 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:00.993125    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wgts9"]
Sep 09 16:21:00.993356 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:00.993333    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-wgts9" podUID="e785d84f-f078-4919-876d-7a60f05035d5" containerName="authorino" containerID="cri-o://488f0ffccadf8aa3bb9b8697a20676a36d4fd53064d4f20d426d55ba99cbb110" gracePeriod=30
Sep 09 16:21:01.252094 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.252065    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-5d6qd"
Sep 09 16:21:01.254531 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.254513    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wgts9"
Sep 09 16:21:01.332824 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.332795    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxk8d\" (UniqueName: \"kubernetes.io/projected/e785d84f-f078-4919-876d-7a60f05035d5-kube-api-access-sxk8d\") pod \"e785d84f-f078-4919-876d-7a60f05035d5\" (UID: \"e785d84f-f078-4919-876d-7a60f05035d5\") "
Sep 09 16:21:01.332959 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.332836    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e496b37-dbaf-4b8f-9bdc-4d049c4cf178-tls-cert\") pod \"1e496b37-dbaf-4b8f-9bdc-4d049c4cf178\" (UID: \"1e496b37-dbaf-4b8f-9bdc-4d049c4cf178\") "
Sep 09 16:21:01.332959 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.332877    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdrkc\" (UniqueName: \"kubernetes.io/projected/1e496b37-dbaf-4b8f-9bdc-4d049c4cf178-kube-api-access-vdrkc\") pod \"1e496b37-dbaf-4b8f-9bdc-4d049c4cf178\" (UID: \"1e496b37-dbaf-4b8f-9bdc-4d049c4cf178\") "
Sep 09 16:21:01.334978 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.334941    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e785d84f-f078-4919-876d-7a60f05035d5-kube-api-access-sxk8d" (OuterVolumeSpecName: "kube-api-access-sxk8d") pod "e785d84f-f078-4919-876d-7a60f05035d5" (UID: "e785d84f-f078-4919-876d-7a60f05035d5"). InnerVolumeSpecName "kube-api-access-sxk8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:21:01.335093 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.335064    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e496b37-dbaf-4b8f-9bdc-4d049c4cf178-kube-api-access-vdrkc" (OuterVolumeSpecName: "kube-api-access-vdrkc") pod "1e496b37-dbaf-4b8f-9bdc-4d049c4cf178" (UID: "1e496b37-dbaf-4b8f-9bdc-4d049c4cf178"). InnerVolumeSpecName "kube-api-access-vdrkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:21:01.341865 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.341840    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e496b37-dbaf-4b8f-9bdc-4d049c4cf178-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "1e496b37-dbaf-4b8f-9bdc-4d049c4cf178" (UID: "1e496b37-dbaf-4b8f-9bdc-4d049c4cf178"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:21:01.434029 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.433978    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxk8d\" (UniqueName: \"kubernetes.io/projected/e785d84f-f078-4919-876d-7a60f05035d5-kube-api-access-sxk8d\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:21:01.434029 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.434022    2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e496b37-dbaf-4b8f-9bdc-4d049c4cf178-tls-cert\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:21:01.434029 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.434037    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdrkc\" (UniqueName: \"kubernetes.io/projected/1e496b37-dbaf-4b8f-9bdc-4d049c4cf178-kube-api-access-vdrkc\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:21:01.956166 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.956125    2576 generic.go:356] "Generic (PLEG): container finished" podID="e785d84f-f078-4919-876d-7a60f05035d5" containerID="488f0ffccadf8aa3bb9b8697a20676a36d4fd53064d4f20d426d55ba99cbb110" exitCode=0
Sep 09 16:21:01.956166 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.956166    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wgts9"
Sep 09 16:21:01.956721 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.956218    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wgts9" event={"ID":"e785d84f-f078-4919-876d-7a60f05035d5","Type":"ContainerDied","Data":"488f0ffccadf8aa3bb9b8697a20676a36d4fd53064d4f20d426d55ba99cbb110"}
Sep 09 16:21:01.956721 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.956256    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wgts9" event={"ID":"e785d84f-f078-4919-876d-7a60f05035d5","Type":"ContainerDied","Data":"7ed97a891c838ea6adb2d56736d5b9423748282c426d8dd9afa6d0c09e9424a7"}
Sep 09 16:21:01.956721 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.956272    2576 scope.go:117] "RemoveContainer" containerID="488f0ffccadf8aa3bb9b8697a20676a36d4fd53064d4f20d426d55ba99cbb110"
Sep 09 16:21:01.957531 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.957507    2576 generic.go:356] "Generic (PLEG): container finished" podID="1e496b37-dbaf-4b8f-9bdc-4d049c4cf178" containerID="b2378cf0910b5d212cd0a62c7cd52c88fb88dc606f64be9aa13890a9ebba6ace" exitCode=0
Sep 09 16:21:01.957635 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.957551    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-5d6qd"
Sep 09 16:21:01.957635 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.957587    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-5d6qd" event={"ID":"1e496b37-dbaf-4b8f-9bdc-4d049c4cf178","Type":"ContainerDied","Data":"b2378cf0910b5d212cd0a62c7cd52c88fb88dc606f64be9aa13890a9ebba6ace"}
Sep 09 16:21:01.957635 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.957618    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-5d6qd" event={"ID":"1e496b37-dbaf-4b8f-9bdc-4d049c4cf178","Type":"ContainerDied","Data":"8aa3c40cf4f17d0bf86bdc4db00193512b5fa77c2706b03ee8818d75216458cf"}
Sep 09 16:21:01.964644 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.964625    2576 scope.go:117] "RemoveContainer" containerID="488f0ffccadf8aa3bb9b8697a20676a36d4fd53064d4f20d426d55ba99cbb110"
Sep 09 16:21:01.965251 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:21:01.964977    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"488f0ffccadf8aa3bb9b8697a20676a36d4fd53064d4f20d426d55ba99cbb110\": container with ID starting with 488f0ffccadf8aa3bb9b8697a20676a36d4fd53064d4f20d426d55ba99cbb110 not found: ID does not exist" containerID="488f0ffccadf8aa3bb9b8697a20676a36d4fd53064d4f20d426d55ba99cbb110"
Sep 09 16:21:01.965251 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.965003    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"488f0ffccadf8aa3bb9b8697a20676a36d4fd53064d4f20d426d55ba99cbb110"} err="failed to get container status \"488f0ffccadf8aa3bb9b8697a20676a36d4fd53064d4f20d426d55ba99cbb110\": rpc error: code = NotFound desc = could not find container \"488f0ffccadf8aa3bb9b8697a20676a36d4fd53064d4f20d426d55ba99cbb110\": container with ID starting with 488f0ffccadf8aa3bb9b8697a20676a36d4fd53064d4f20d426d55ba99cbb110 not found: ID does not exist"
Sep 09 16:21:01.965251 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.965019    2576 scope.go:117] "RemoveContainer" containerID="b2378cf0910b5d212cd0a62c7cd52c88fb88dc606f64be9aa13890a9ebba6ace"
Sep 09 16:21:01.972662 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.972647    2576 scope.go:117] "RemoveContainer" containerID="b2378cf0910b5d212cd0a62c7cd52c88fb88dc606f64be9aa13890a9ebba6ace"
Sep 09 16:21:01.972959 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:21:01.972938    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2378cf0910b5d212cd0a62c7cd52c88fb88dc606f64be9aa13890a9ebba6ace\": container with ID starting with b2378cf0910b5d212cd0a62c7cd52c88fb88dc606f64be9aa13890a9ebba6ace not found: ID does not exist" containerID="b2378cf0910b5d212cd0a62c7cd52c88fb88dc606f64be9aa13890a9ebba6ace"
Sep 09 16:21:01.973013 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.972969    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2378cf0910b5d212cd0a62c7cd52c88fb88dc606f64be9aa13890a9ebba6ace"} err="failed to get container status \"b2378cf0910b5d212cd0a62c7cd52c88fb88dc606f64be9aa13890a9ebba6ace\": rpc error: code = NotFound desc = could not find container \"b2378cf0910b5d212cd0a62c7cd52c88fb88dc606f64be9aa13890a9ebba6ace\": container with ID starting with b2378cf0910b5d212cd0a62c7cd52c88fb88dc606f64be9aa13890a9ebba6ace not found: ID does not exist"
Sep 09 16:21:01.978224 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.978203    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5d6qd"]
Sep 09 16:21:01.983517 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.983498    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5d6qd"]
Sep 09 16:21:01.992605 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.992583    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wgts9"]
Sep 09 16:21:01.994348 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:01.994330    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wgts9"]
Sep 09 16:21:02.552260 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:02.552230    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e496b37-dbaf-4b8f-9bdc-4d049c4cf178" path="/var/lib/kubelet/pods/1e496b37-dbaf-4b8f-9bdc-4d049c4cf178/volumes"
Sep 09 16:21:02.552524 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:02.552512    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e785d84f-f078-4919-876d-7a60f05035d5" path="/var/lib/kubelet/pods/e785d84f-f078-4919-876d-7a60f05035d5/volumes"
Sep 09 16:21:44.255140 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.255108    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z"]
Sep 09 16:21:44.255592 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.255348    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e785d84f-f078-4919-876d-7a60f05035d5" containerName="authorino"
Sep 09 16:21:44.255592 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.255358    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e785d84f-f078-4919-876d-7a60f05035d5" containerName="authorino"
Sep 09 16:21:44.255592 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.255370    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e496b37-dbaf-4b8f-9bdc-4d049c4cf178" containerName="authorino"
Sep 09 16:21:44.255592 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.255375    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e496b37-dbaf-4b8f-9bdc-4d049c4cf178" containerName="authorino"
Sep 09 16:21:44.255592 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.255419    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e785d84f-f078-4919-876d-7a60f05035d5" containerName="authorino"
Sep 09 16:21:44.255592 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.255427    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e496b37-dbaf-4b8f-9bdc-4d049c4cf178" containerName="authorino"
Sep 09 16:21:44.258179 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.258163    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z"
Sep 09 16:21:44.260503 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.260476    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 16:21:44.260609 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.260479    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jt2jt\""
Sep 09 16:21:44.268488 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.268460    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z"]
Sep 09 16:21:44.289674 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.289644    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-s76z5"]
Sep 09 16:21:44.292594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.292579    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-s76z5"
Sep 09 16:21:44.304185 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.304158    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-s76z5"]
Sep 09 16:21:44.358728 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.358678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d34adc-3182-4feb-acba-41b98f7c6a8b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-bjd4z\" (UID: \"24d34adc-3182-4feb-acba-41b98f7c6a8b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z"
Sep 09 16:21:44.358903 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.358735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86m6\" (UniqueName: \"kubernetes.io/projected/24d34adc-3182-4feb-acba-41b98f7c6a8b-kube-api-access-b86m6\") pod \"payload-pre-processing-59fbc47bf8-bjd4z\" (UID: \"24d34adc-3182-4feb-acba-41b98f7c6a8b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z"
Sep 09 16:21:44.459145 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.459111    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d34adc-3182-4feb-acba-41b98f7c6a8b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-bjd4z\" (UID: \"24d34adc-3182-4feb-acba-41b98f7c6a8b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z"
Sep 09 16:21:44.459381 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.459363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48mnq\" (UniqueName: \"kubernetes.io/projected/6bcd5da5-f6c4-46aa-87dd-21844af21310-kube-api-access-48mnq\") pod \"payload-processing-5b54b7fbb7-s76z5\" (UID: \"6bcd5da5-f6c4-46aa-87dd-21844af21310\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-s76z5"
Sep 09 16:21:44.459519 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.459505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86m6\" (UniqueName: \"kubernetes.io/projected/24d34adc-3182-4feb-acba-41b98f7c6a8b-kube-api-access-b86m6\") pod \"payload-pre-processing-59fbc47bf8-bjd4z\" (UID: \"24d34adc-3182-4feb-acba-41b98f7c6a8b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z"
Sep 09 16:21:44.459752 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.459736    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcd5da5-f6c4-46aa-87dd-21844af21310-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-s76z5\" (UID: \"6bcd5da5-f6c4-46aa-87dd-21844af21310\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-s76z5"
Sep 09 16:21:44.461633 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.461611    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 16:21:44.470390 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.470360    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86m6\" (UniqueName: \"kubernetes.io/projected/24d34adc-3182-4feb-acba-41b98f7c6a8b-kube-api-access-b86m6\") pod \"payload-pre-processing-59fbc47bf8-bjd4z\" (UID: \"24d34adc-3182-4feb-acba-41b98f7c6a8b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z"
Sep 09 16:21:44.470505 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.470489    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d34adc-3182-4feb-acba-41b98f7c6a8b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-bjd4z\" (UID: \"24d34adc-3182-4feb-acba-41b98f7c6a8b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z"
Sep 09 16:21:44.561026 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.560962    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48mnq\" (UniqueName: \"kubernetes.io/projected/6bcd5da5-f6c4-46aa-87dd-21844af21310-kube-api-access-48mnq\") pod \"payload-processing-5b54b7fbb7-s76z5\" (UID: \"6bcd5da5-f6c4-46aa-87dd-21844af21310\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-s76z5"
Sep 09 16:21:44.561168 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.561049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcd5da5-f6c4-46aa-87dd-21844af21310-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-s76z5\" (UID: \"6bcd5da5-f6c4-46aa-87dd-21844af21310\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-s76z5"
Sep 09 16:21:44.561680 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.561661    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcd5da5-f6c4-46aa-87dd-21844af21310-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-s76z5\" (UID: \"6bcd5da5-f6c4-46aa-87dd-21844af21310\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-s76z5"
Sep 09 16:21:44.569688 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.569657    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48mnq\" (UniqueName: \"kubernetes.io/projected/6bcd5da5-f6c4-46aa-87dd-21844af21310-kube-api-access-48mnq\") pod \"payload-processing-5b54b7fbb7-s76z5\" (UID: \"6bcd5da5-f6c4-46aa-87dd-21844af21310\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-s76z5"
Sep 09 16:21:44.569862 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.569844    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jt2jt\""
Sep 09 16:21:44.578777 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.578746    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z"
Sep 09 16:21:44.602681 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.602654    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-s76z5"
Sep 09 16:21:44.862570 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.862498    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z"]
Sep 09 16:21:44.866005 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:21:44.865975    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d34adc_3182_4feb_acba_41b98f7c6a8b.slice/crio-b4d1731596e9692fe1e4f15e4ae0e8d66aad71dc1184d14206dfdcfb32c68181 WatchSource:0}: Error finding container b4d1731596e9692fe1e4f15e4ae0e8d66aad71dc1184d14206dfdcfb32c68181: Status 404 returned error can't find the container with id b4d1731596e9692fe1e4f15e4ae0e8d66aad71dc1184d14206dfdcfb32c68181
Sep 09 16:21:44.889328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:44.889305    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-s76z5"]
Sep 09 16:21:44.893128 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:21:44.893093    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcd5da5_f6c4_46aa_87dd_21844af21310.slice/crio-5ff4e46168217bc39a583a0ad593cb71fc46c07d4e8fdc57c88aeda40d6a45b0 WatchSource:0}: Error finding container 5ff4e46168217bc39a583a0ad593cb71fc46c07d4e8fdc57c88aeda40d6a45b0: Status 404 returned error can't find the container with id 5ff4e46168217bc39a583a0ad593cb71fc46c07d4e8fdc57c88aeda40d6a45b0
Sep 09 16:21:45.115834 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:45.115730    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-s76z5" event={"ID":"6bcd5da5-f6c4-46aa-87dd-21844af21310","Type":"ContainerStarted","Data":"5ff4e46168217bc39a583a0ad593cb71fc46c07d4e8fdc57c88aeda40d6a45b0"}
Sep 09 16:21:45.116836 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:45.116810    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z" event={"ID":"24d34adc-3182-4feb-acba-41b98f7c6a8b","Type":"ContainerStarted","Data":"b4d1731596e9692fe1e4f15e4ae0e8d66aad71dc1184d14206dfdcfb32c68181"}
Sep 09 16:21:48.130679 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:48.130638    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z" event={"ID":"24d34adc-3182-4feb-acba-41b98f7c6a8b","Type":"ContainerStarted","Data":"331c0845bb6b8bee1bb96b22c0fb006e7c62e5a37b5bce8520f78ce6729022d0"}
Sep 09 16:21:48.131127 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:48.130802    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z"
Sep 09 16:21:48.132201 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:48.132178    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-s76z5" event={"ID":"6bcd5da5-f6c4-46aa-87dd-21844af21310","Type":"ContainerStarted","Data":"fd6622b90111e7450e2b4c2449b20c1cbc605ee28a28f88442cbe49fdc5559e9"}
Sep 09 16:21:48.132309 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:48.132288    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-s76z5"
Sep 09 16:21:48.146355 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:48.146308    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z" podStartSLOduration=1.528946543 podStartE2EDuration="4.146291613s" totalImagesPullingTime="2.61734507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:21:48.145864342 +0000 UTC m=+784.081482782" watchObservedRunningTime="2026-09-09 16:21:48.146291613 +0000 UTC m=+784.081910033"
Sep 09 16:21:48.160758 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:48.160720    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-s76z5" podStartSLOduration=1.565018851 podStartE2EDuration="4.160691444s" totalImagesPullingTime="2.595672593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:21:48.159755404 +0000 UTC m=+784.095373841" watchObservedRunningTime="2026-09-09 16:21:48.160691444 +0000 UTC m=+784.096309861"
Sep 09 16:21:59.138684 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:59.138652    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bjd4z"
Sep 09 16:21:59.139189 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:21:59.138870    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-s76z5"
Sep 09 16:22:42.002982 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.002946    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"]
Sep 09 16:22:42.006671 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.006645    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.010981 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.010954    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 16:22:42.011124 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.011002    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 16:22:42.011124 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.011039    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m45r2\""
Sep 09 16:22:42.011124 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.011007    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 16:22:42.015118 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.015105    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"]
Sep 09 16:22:42.073027 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.072987    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5acf5f60-eccf-48c5-9002-eb668daa5a4b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.073027 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.073028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5acf5f60-eccf-48c5-9002-eb668daa5a4b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.073240 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.073058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5acf5f60-eccf-48c5-9002-eb668daa5a4b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.073240 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.073095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5acf5f60-eccf-48c5-9002-eb668daa5a4b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.073240 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.073112    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5acf5f60-eccf-48c5-9002-eb668daa5a4b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.073240 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.073129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zkjv\" (UniqueName: \"kubernetes.io/projected/5acf5f60-eccf-48c5-9002-eb668daa5a4b-kube-api-access-6zkjv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.174335 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.174306    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5acf5f60-eccf-48c5-9002-eb668daa5a4b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.174335 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.174340    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5acf5f60-eccf-48c5-9002-eb668daa5a4b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.174536 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.174358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5acf5f60-eccf-48c5-9002-eb668daa5a4b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.174536 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.174426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5acf5f60-eccf-48c5-9002-eb668daa5a4b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.174536 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.174457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5acf5f60-eccf-48c5-9002-eb668daa5a4b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.174536 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.174478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zkjv\" (UniqueName: \"kubernetes.io/projected/5acf5f60-eccf-48c5-9002-eb668daa5a4b-kube-api-access-6zkjv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.174756 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.174734    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5acf5f60-eccf-48c5-9002-eb668daa5a4b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.174831 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.174811    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5acf5f60-eccf-48c5-9002-eb668daa5a4b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.174886 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.174852    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5acf5f60-eccf-48c5-9002-eb668daa5a4b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.176829 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.176811    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5acf5f60-eccf-48c5-9002-eb668daa5a4b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.177064 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.177046    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5acf5f60-eccf-48c5-9002-eb668daa5a4b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.182575 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.182553    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zkjv\" (UniqueName: \"kubernetes.io/projected/5acf5f60-eccf-48c5-9002-eb668daa5a4b-kube-api-access-6zkjv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gnnxn\" (UID: \"5acf5f60-eccf-48c5-9002-eb668daa5a4b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.318803 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.318721    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:42.588361 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:42.588339    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"]
Sep 09 16:22:42.590950 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:22:42.590923    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5acf5f60_eccf_48c5_9002_eb668daa5a4b.slice/crio-8b216ba038559c7092f31196c23364fba0895a65b4dcfc192cb8239b3f78ee0b WatchSource:0}: Error finding container 8b216ba038559c7092f31196c23364fba0895a65b4dcfc192cb8239b3f78ee0b: Status 404 returned error can't find the container with id 8b216ba038559c7092f31196c23364fba0895a65b4dcfc192cb8239b3f78ee0b
Sep 09 16:22:43.342870 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:43.342825    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn" event={"ID":"5acf5f60-eccf-48c5-9002-eb668daa5a4b","Type":"ContainerStarted","Data":"8b216ba038559c7092f31196c23364fba0895a65b4dcfc192cb8239b3f78ee0b"}
Sep 09 16:22:44.348325 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:44.348288    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn" event={"ID":"5acf5f60-eccf-48c5-9002-eb668daa5a4b","Type":"ContainerStarted","Data":"52c8a9f033abb9884b3021430f1a7d9fe4afb1ee8484e2437da36e1ae5c93baf"}
Sep 09 16:22:44.365792 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:44.365731    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn" podStartSLOduration=2.353061905 podStartE2EDuration="3.365710531s" totalImagesPullingTime="1.012648626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:22:44.364212426 +0000 UTC m=+840.299830852" watchObservedRunningTime="2026-09-09 16:22:44.365710531 +0000 UTC m=+840.301328941"
Sep 09 16:22:52.319549 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:52.319509    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:52.320342 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:52.319574    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:52.331900 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:52.331876    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:52.390010 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:52.389984    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gnnxn"
Sep 09 16:22:54.208860 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.208828    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"]
Sep 09 16:22:54.211104 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.211086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.213718 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.213680    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 16:22:54.222642 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.222619    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"]
Sep 09 16:22:54.371938 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.371904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c88da762-3854-4b2b-b628-4297e4b1de46-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.372118 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.371948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c88da762-3854-4b2b-b628-4297e4b1de46-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.372118 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.372017    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c88da762-3854-4b2b-b628-4297e4b1de46-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.372118 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.372054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsz24\" (UniqueName: \"kubernetes.io/projected/c88da762-3854-4b2b-b628-4297e4b1de46-kube-api-access-wsz24\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.372118 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.372103    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c88da762-3854-4b2b-b628-4297e4b1de46-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.372118 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.372119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c88da762-3854-4b2b-b628-4297e4b1de46-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.473381 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.473349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c88da762-3854-4b2b-b628-4297e4b1de46-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.473530 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.473387    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c88da762-3854-4b2b-b628-4297e4b1de46-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.473530 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.473409    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c88da762-3854-4b2b-b628-4297e4b1de46-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.473530 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.473439    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c88da762-3854-4b2b-b628-4297e4b1de46-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.473530 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.473486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c88da762-3854-4b2b-b628-4297e4b1de46-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.473530 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.473516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsz24\" (UniqueName: \"kubernetes.io/projected/c88da762-3854-4b2b-b628-4297e4b1de46-kube-api-access-wsz24\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.473858 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.473822    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c88da762-3854-4b2b-b628-4297e4b1de46-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.473858 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.473835    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c88da762-3854-4b2b-b628-4297e4b1de46-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.474018 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.473880    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c88da762-3854-4b2b-b628-4297e4b1de46-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.475881 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.475856    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c88da762-3854-4b2b-b628-4297e4b1de46-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.476064 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.476043    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c88da762-3854-4b2b-b628-4297e4b1de46-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.482506 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.482484    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsz24\" (UniqueName: \"kubernetes.io/projected/c88da762-3854-4b2b-b628-4297e4b1de46-kube-api-access-wsz24\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc\" (UID: \"c88da762-3854-4b2b-b628-4297e4b1de46\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.521171 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.521145    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:22:54.787749 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:54.787725    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"]
Sep 09 16:22:54.790071 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:22:54.790048    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88da762_3854_4b2b_b628_4297e4b1de46.slice/crio-cf8c03b425b52b35af6f162ad1b799dcaa7a232abf1c4e516c70bb9fcfb26701 WatchSource:0}: Error finding container cf8c03b425b52b35af6f162ad1b799dcaa7a232abf1c4e516c70bb9fcfb26701: Status 404 returned error can't find the container with id cf8c03b425b52b35af6f162ad1b799dcaa7a232abf1c4e516c70bb9fcfb26701
Sep 09 16:22:55.395414 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:55.395377    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc" event={"ID":"c88da762-3854-4b2b-b628-4297e4b1de46","Type":"ContainerStarted","Data":"bf8a587d1dfcd704db8d0d8003d6b5e6c4add030911ac9a4ea8b3e641c723d51"}
Sep 09 16:22:55.395821 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:55.395434    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc" event={"ID":"c88da762-3854-4b2b-b628-4297e4b1de46","Type":"ContainerStarted","Data":"cf8c03b425b52b35af6f162ad1b799dcaa7a232abf1c4e516c70bb9fcfb26701"}
Sep 09 16:22:55.412022 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:22:55.411972    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc" podStartSLOduration=1.23288269 podStartE2EDuration="1.411957492s" totalImagesPullingTime="179.074802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:22:55.410325676 +0000 UTC m=+851.345944104" watchObservedRunningTime="2026-09-09 16:22:55.411957492 +0000 UTC m=+851.347575948"
Sep 09 16:23:04.522077 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:04.522045    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:23:04.522480 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:04.522144    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:23:04.534437 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:04.534417    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:23:05.446805 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:05.446774    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc"
Sep 09 16:23:09.697483 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.697446    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"]
Sep 09 16:23:09.705491 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.705461    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.708194 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.708155    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 16:23:09.711667 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.711639    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"]
Sep 09 16:23:09.799510 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.799474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02ab45f4-89d6-47b1-9b4d-99184d95b633-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.799510 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.799512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02ab45f4-89d6-47b1-9b4d-99184d95b633-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.799768 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.799542    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24ljw\" (UniqueName: \"kubernetes.io/projected/02ab45f4-89d6-47b1-9b4d-99184d95b633-kube-api-access-24ljw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.799768 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.799600    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02ab45f4-89d6-47b1-9b4d-99184d95b633-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.799768 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.799622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02ab45f4-89d6-47b1-9b4d-99184d95b633-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.799768 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.799674    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ab45f4-89d6-47b1-9b4d-99184d95b633-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.900419 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.900384    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02ab45f4-89d6-47b1-9b4d-99184d95b633-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.900419 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.900424    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02ab45f4-89d6-47b1-9b4d-99184d95b633-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.900586 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.900450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24ljw\" (UniqueName: \"kubernetes.io/projected/02ab45f4-89d6-47b1-9b4d-99184d95b633-kube-api-access-24ljw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.900639 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.900577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02ab45f4-89d6-47b1-9b4d-99184d95b633-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.900639 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.900618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02ab45f4-89d6-47b1-9b4d-99184d95b633-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.900775 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.900665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ab45f4-89d6-47b1-9b4d-99184d95b633-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.901002 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.900974    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02ab45f4-89d6-47b1-9b4d-99184d95b633-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.901103 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.901035    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ab45f4-89d6-47b1-9b4d-99184d95b633-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.901103 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.901051    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02ab45f4-89d6-47b1-9b4d-99184d95b633-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.902897 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.902865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02ab45f4-89d6-47b1-9b4d-99184d95b633-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.903084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.903065    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02ab45f4-89d6-47b1-9b4d-99184d95b633-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:09.909355 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:09.909336    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24ljw\" (UniqueName: \"kubernetes.io/projected/02ab45f4-89d6-47b1-9b4d-99184d95b633-kube-api-access-24ljw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6\" (UID: \"02ab45f4-89d6-47b1-9b4d-99184d95b633\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:10.016316 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:10.016269    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:10.286099 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:10.285985    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"]
Sep 09 16:23:10.458262 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:10.458228    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6" event={"ID":"02ab45f4-89d6-47b1-9b4d-99184d95b633","Type":"ContainerStarted","Data":"f249e0ef1f109449e94b967231ad257d72e9dd5da7b0789e77b8b87481f25bcc"}
Sep 09 16:23:11.463617 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:11.463582    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6" event={"ID":"02ab45f4-89d6-47b1-9b4d-99184d95b633","Type":"ContainerStarted","Data":"c7c5be576c6323578ea2983a851490f5b9ba9a340363bca501f5c4cf65e62248"}
Sep 09 16:23:11.481953 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:11.481909    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6" podStartSLOduration=2.313417569 podStartE2EDuration="2.481895189s" totalImagesPullingTime="168.47762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:23:11.479755041 +0000 UTC m=+867.415373478" watchObservedRunningTime="2026-09-09 16:23:11.481895189 +0000 UTC m=+867.417513604"
Sep 09 16:23:20.016749 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:20.016714    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:20.016749 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:20.016758    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:20.029083 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:20.029055    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:23:20.510987 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:23:20.510959    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6"
Sep 09 16:24:36.283516 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:36.283485    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f6f59dbd4-m82tr"]
Sep 09 16:24:36.286591 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:36.286571    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f6f59dbd4-m82tr"
Sep 09 16:24:36.298179 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:36.298153    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f6f59dbd4-m82tr"]
Sep 09 16:24:36.402041 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:36.402009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41cb6b67-baf5-4685-a796-f4c08b233498-tls-cert\") pod \"authorino-7f6f59dbd4-m82tr\" (UID: \"41cb6b67-baf5-4685-a796-f4c08b233498\") " pod="kuadrant-system/authorino-7f6f59dbd4-m82tr"
Sep 09 16:24:36.402195 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:36.402066    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgrdf\" (UniqueName: \"kubernetes.io/projected/41cb6b67-baf5-4685-a796-f4c08b233498-kube-api-access-mgrdf\") pod \"authorino-7f6f59dbd4-m82tr\" (UID: \"41cb6b67-baf5-4685-a796-f4c08b233498\") " pod="kuadrant-system/authorino-7f6f59dbd4-m82tr"
Sep 09 16:24:36.502773 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:36.502686    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41cb6b67-baf5-4685-a796-f4c08b233498-tls-cert\") pod \"authorino-7f6f59dbd4-m82tr\" (UID: \"41cb6b67-baf5-4685-a796-f4c08b233498\") " pod="kuadrant-system/authorino-7f6f59dbd4-m82tr"
Sep 09 16:24:36.502933 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:36.502800    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgrdf\" (UniqueName: \"kubernetes.io/projected/41cb6b67-baf5-4685-a796-f4c08b233498-kube-api-access-mgrdf\") pod \"authorino-7f6f59dbd4-m82tr\" (UID: \"41cb6b67-baf5-4685-a796-f4c08b233498\") " pod="kuadrant-system/authorino-7f6f59dbd4-m82tr"
Sep 09 16:24:36.509079 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:36.509035    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41cb6b67-baf5-4685-a796-f4c08b233498-tls-cert\") pod \"authorino-7f6f59dbd4-m82tr\" (UID: \"41cb6b67-baf5-4685-a796-f4c08b233498\") " pod="kuadrant-system/authorino-7f6f59dbd4-m82tr"
Sep 09 16:24:36.511905 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:36.511883    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgrdf\" (UniqueName: \"kubernetes.io/projected/41cb6b67-baf5-4685-a796-f4c08b233498-kube-api-access-mgrdf\") pod \"authorino-7f6f59dbd4-m82tr\" (UID: \"41cb6b67-baf5-4685-a796-f4c08b233498\") " pod="kuadrant-system/authorino-7f6f59dbd4-m82tr"
Sep 09 16:24:36.595869 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:36.595779    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f6f59dbd4-m82tr"
Sep 09 16:24:36.867354 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:36.867269    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f6f59dbd4-m82tr"]
Sep 09 16:24:36.871786 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:24:36.871760    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41cb6b67_baf5_4685_a796_f4c08b233498.slice/crio-86623dec34b29be8701cfb43b9fe2c9891a8d9bd175bbb13af75039ec43f0fda WatchSource:0}: Error finding container 86623dec34b29be8701cfb43b9fe2c9891a8d9bd175bbb13af75039ec43f0fda: Status 404 returned error can't find the container with id 86623dec34b29be8701cfb43b9fe2c9891a8d9bd175bbb13af75039ec43f0fda
Sep 09 16:24:37.806493 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:37.806450    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f6f59dbd4-m82tr" event={"ID":"41cb6b67-baf5-4685-a796-f4c08b233498","Type":"ContainerStarted","Data":"2ff70dcfebf44fe2b765a220b1264c664ca0ec6f4486915c70dcb7ee306f575c"}
Sep 09 16:24:37.806887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:37.806501    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f6f59dbd4-m82tr" event={"ID":"41cb6b67-baf5-4685-a796-f4c08b233498","Type":"ContainerStarted","Data":"86623dec34b29be8701cfb43b9fe2c9891a8d9bd175bbb13af75039ec43f0fda"}
Sep 09 16:24:37.825975 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:37.825899    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7f6f59dbd4-m82tr" podStartSLOduration=1.401103427 podStartE2EDuration="1.825875433s" totalImagesPullingTime="424.772006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:24:37.823490112 +0000 UTC m=+953.759108528" watchObservedRunningTime="2026-09-09 16:24:37.825875433 +0000 UTC m=+953.761493853"
Sep 09 16:24:37.849151 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:37.849115    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7576868b45-r74x2"]
Sep 09 16:24:37.849387 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:37.849360    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7576868b45-r74x2" podUID="35659680-4927-4e1d-b21a-479365e0f2cf" containerName="authorino" containerID="cri-o://d75c84b0a930334d7294f21786bc18dcf1601efc5d0346897fbabb2c7b92371e" gracePeriod=30
Sep 09 16:24:38.105612 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.105589    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7576868b45-r74x2"
Sep 09 16:24:38.216288 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.216250    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35659680-4927-4e1d-b21a-479365e0f2cf-tls-cert\") pod \"35659680-4927-4e1d-b21a-479365e0f2cf\" (UID: \"35659680-4927-4e1d-b21a-479365e0f2cf\") "
Sep 09 16:24:38.216478 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.216306    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhkcl\" (UniqueName: \"kubernetes.io/projected/35659680-4927-4e1d-b21a-479365e0f2cf-kube-api-access-lhkcl\") pod \"35659680-4927-4e1d-b21a-479365e0f2cf\" (UID: \"35659680-4927-4e1d-b21a-479365e0f2cf\") "
Sep 09 16:24:38.218624 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.218590    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35659680-4927-4e1d-b21a-479365e0f2cf-kube-api-access-lhkcl" (OuterVolumeSpecName: "kube-api-access-lhkcl") pod "35659680-4927-4e1d-b21a-479365e0f2cf" (UID: "35659680-4927-4e1d-b21a-479365e0f2cf"). InnerVolumeSpecName "kube-api-access-lhkcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:24:38.225779 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.225690    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35659680-4927-4e1d-b21a-479365e0f2cf-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "35659680-4927-4e1d-b21a-479365e0f2cf" (UID: "35659680-4927-4e1d-b21a-479365e0f2cf"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:24:38.317368 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.317324    2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35659680-4927-4e1d-b21a-479365e0f2cf-tls-cert\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:24:38.317368 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.317370    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhkcl\" (UniqueName: \"kubernetes.io/projected/35659680-4927-4e1d-b21a-479365e0f2cf-kube-api-access-lhkcl\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:24:38.812293 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.812203    2576 generic.go:356] "Generic (PLEG): container finished" podID="35659680-4927-4e1d-b21a-479365e0f2cf" containerID="d75c84b0a930334d7294f21786bc18dcf1601efc5d0346897fbabb2c7b92371e" exitCode=0
Sep 09 16:24:38.812293 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.812281    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7576868b45-r74x2"
Sep 09 16:24:38.812763 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.812302    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7576868b45-r74x2" event={"ID":"35659680-4927-4e1d-b21a-479365e0f2cf","Type":"ContainerDied","Data":"d75c84b0a930334d7294f21786bc18dcf1601efc5d0346897fbabb2c7b92371e"}
Sep 09 16:24:38.812763 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.812345    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7576868b45-r74x2" event={"ID":"35659680-4927-4e1d-b21a-479365e0f2cf","Type":"ContainerDied","Data":"dc871999f841c22bea95df1f17c3108f55cc7bf9629b38eb7bb1f12f854853ad"}
Sep 09 16:24:38.812763 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.812362    2576 scope.go:117] "RemoveContainer" containerID="d75c84b0a930334d7294f21786bc18dcf1601efc5d0346897fbabb2c7b92371e"
Sep 09 16:24:38.820762 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.820737    2576 scope.go:117] "RemoveContainer" containerID="d75c84b0a930334d7294f21786bc18dcf1601efc5d0346897fbabb2c7b92371e"
Sep 09 16:24:38.821080 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:24:38.821054    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d75c84b0a930334d7294f21786bc18dcf1601efc5d0346897fbabb2c7b92371e\": container with ID starting with d75c84b0a930334d7294f21786bc18dcf1601efc5d0346897fbabb2c7b92371e not found: ID does not exist" containerID="d75c84b0a930334d7294f21786bc18dcf1601efc5d0346897fbabb2c7b92371e"
Sep 09 16:24:38.821143 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.821088    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75c84b0a930334d7294f21786bc18dcf1601efc5d0346897fbabb2c7b92371e"} err="failed to get container status \"d75c84b0a930334d7294f21786bc18dcf1601efc5d0346897fbabb2c7b92371e\": rpc error: code = NotFound desc = could not find container \"d75c84b0a930334d7294f21786bc18dcf1601efc5d0346897fbabb2c7b92371e\": container with ID starting with d75c84b0a930334d7294f21786bc18dcf1601efc5d0346897fbabb2c7b92371e not found: ID does not exist"
Sep 09 16:24:38.833012 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.832985    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7576868b45-r74x2"]
Sep 09 16:24:38.836520 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:38.836499    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7576868b45-r74x2"]
Sep 09 16:24:40.552763 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:40.552730    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35659680-4927-4e1d-b21a-479365e0f2cf" path="/var/lib/kubelet/pods/35659680-4927-4e1d-b21a-479365e0f2cf/volumes"
Sep 09 16:24:58.380089 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:58.380036    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-140146116713776"]
Sep 09 16:24:58.380507 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:58.380316    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35659680-4927-4e1d-b21a-479365e0f2cf" containerName="authorino"
Sep 09 16:24:58.380507 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:58.380327    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="35659680-4927-4e1d-b21a-479365e0f2cf" containerName="authorino"
Sep 09 16:24:58.380507 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:58.380373    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="35659680-4927-4e1d-b21a-479365e0f2cf" containerName="authorino"
Sep 09 16:24:58.384568 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:58.384551    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:24:58.387051 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:58.387028    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bzk7m\""
Sep 09 16:24:58.390038 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:58.390017    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-140146116713776"]
Sep 09 16:24:58.482434 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:58.482392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rctwn\" (UniqueName: \"kubernetes.io/projected/d0503b9c-dd83-4534-b2de-472b03c06371-kube-api-access-rctwn\") pod \"test-curl-7697-140146116713776\" (UID: \"d0503b9c-dd83-4534-b2de-472b03c06371\") " pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:24:58.583543 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:58.583503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rctwn\" (UniqueName: \"kubernetes.io/projected/d0503b9c-dd83-4534-b2de-472b03c06371-kube-api-access-rctwn\") pod \"test-curl-7697-140146116713776\" (UID: \"d0503b9c-dd83-4534-b2de-472b03c06371\") " pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:24:58.591097 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:58.591063    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rctwn\" (UniqueName: \"kubernetes.io/projected/d0503b9c-dd83-4534-b2de-472b03c06371-kube-api-access-rctwn\") pod \"test-curl-7697-140146116713776\" (UID: \"d0503b9c-dd83-4534-b2de-472b03c06371\") " pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:24:58.694566 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:58.694471    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:24:58.964303 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:58.964178    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-140146116713776"]
Sep 09 16:24:58.966760 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:24:58.966732    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0503b9c_dd83_4534_b2de_472b03c06371.slice/crio-1a3ca4f16c21b90bdf3cf6c92160cb374696b063930161d75fde2c0942969fe7 WatchSource:0}: Error finding container 1a3ca4f16c21b90bdf3cf6c92160cb374696b063930161d75fde2c0942969fe7: Status 404 returned error can't find the container with id 1a3ca4f16c21b90bdf3cf6c92160cb374696b063930161d75fde2c0942969fe7
Sep 09 16:24:59.895437 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:24:59.895404    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-140146116713776" event={"ID":"d0503b9c-dd83-4534-b2de-472b03c06371","Type":"ContainerStarted","Data":"1a3ca4f16c21b90bdf3cf6c92160cb374696b063930161d75fde2c0942969fe7"}
Sep 09 16:25:01.713173 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.713084    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"]
Sep 09 16:25:01.716320 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.716300    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"
Sep 09 16:25:01.718591 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.718569    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6d44b6\""
Sep 09 16:25:01.728330 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.728308    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"]
Sep 09 16:25:01.740372 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.740337    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"]
Sep 09 16:25:01.743373 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.743359    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"
Sep 09 16:25:01.745605 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.745586    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6d44b6-dockercfg-j948d\""
Sep 09 16:25:01.753293 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.753258    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"]
Sep 09 16:25:01.812376 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.812342    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dlsn\" (UniqueName: \"kubernetes.io/projected/f55f996a-d2aa-4fc8-a38d-798a7053e0b5-kube-api-access-2dlsn\") pod \"payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l\" (UID: \"f55f996a-d2aa-4fc8-a38d-798a7053e0b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"
Sep 09 16:25:01.812521 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.812393    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55f996a-d2aa-4fc8-a38d-798a7053e0b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l\" (UID: \"f55f996a-d2aa-4fc8-a38d-798a7053e0b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"
Sep 09 16:25:01.910083 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.910045    2576 generic.go:356] "Generic (PLEG): container finished" podID="d0503b9c-dd83-4534-b2de-472b03c06371" containerID="ef57c15f538e834a50d47729a1d582eb0352ed1f747688a8309aeefbed9f35d1" exitCode=0
Sep 09 16:25:01.910252 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.910094    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-140146116713776" event={"ID":"d0503b9c-dd83-4534-b2de-472b03c06371","Type":"ContainerDied","Data":"ef57c15f538e834a50d47729a1d582eb0352ed1f747688a8309aeefbed9f35d1"}
Sep 09 16:25:01.913074 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.913050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55f996a-d2aa-4fc8-a38d-798a7053e0b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l\" (UID: \"f55f996a-d2aa-4fc8-a38d-798a7053e0b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"
Sep 09 16:25:01.913130 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.913092    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf4rn\" (UniqueName: \"kubernetes.io/projected/03b12307-d8e3-4da6-8c4a-494b3a3b0862-kube-api-access-xf4rn\") pod \"payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj\" (UID: \"03b12307-d8e3-4da6-8c4a-494b3a3b0862\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"
Sep 09 16:25:01.913172 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.913154    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b12307-d8e3-4da6-8c4a-494b3a3b0862-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj\" (UID: \"03b12307-d8e3-4da6-8c4a-494b3a3b0862\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"
Sep 09 16:25:01.913224 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.913210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dlsn\" (UniqueName: \"kubernetes.io/projected/f55f996a-d2aa-4fc8-a38d-798a7053e0b5-kube-api-access-2dlsn\") pod \"payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l\" (UID: \"f55f996a-d2aa-4fc8-a38d-798a7053e0b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"
Sep 09 16:25:01.913938 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.913920    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55f996a-d2aa-4fc8-a38d-798a7053e0b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l\" (UID: \"f55f996a-d2aa-4fc8-a38d-798a7053e0b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"
Sep 09 16:25:01.920910 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:01.920890    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dlsn\" (UniqueName: \"kubernetes.io/projected/f55f996a-d2aa-4fc8-a38d-798a7053e0b5-kube-api-access-2dlsn\") pod \"payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l\" (UID: \"f55f996a-d2aa-4fc8-a38d-798a7053e0b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"
Sep 09 16:25:02.014288 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.014247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf4rn\" (UniqueName: \"kubernetes.io/projected/03b12307-d8e3-4da6-8c4a-494b3a3b0862-kube-api-access-xf4rn\") pod \"payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj\" (UID: \"03b12307-d8e3-4da6-8c4a-494b3a3b0862\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"
Sep 09 16:25:02.014506 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.014406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b12307-d8e3-4da6-8c4a-494b3a3b0862-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj\" (UID: \"03b12307-d8e3-4da6-8c4a-494b3a3b0862\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"
Sep 09 16:25:02.015181 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.015161    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b12307-d8e3-4da6-8c4a-494b3a3b0862-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj\" (UID: \"03b12307-d8e3-4da6-8c4a-494b3a3b0862\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"
Sep 09 16:25:02.022047 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.022022    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf4rn\" (UniqueName: \"kubernetes.io/projected/03b12307-d8e3-4da6-8c4a-494b3a3b0862-kube-api-access-xf4rn\") pod \"payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj\" (UID: \"03b12307-d8e3-4da6-8c4a-494b3a3b0862\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"
Sep 09 16:25:02.025790 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.025774    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"
Sep 09 16:25:02.054161 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.054131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"
Sep 09 16:25:02.303545 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.303517    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"]
Sep 09 16:25:02.305752 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:25:02.305722    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55f996a_d2aa_4fc8_a38d_798a7053e0b5.slice/crio-445b3d099e9d20d6173e2638ad2df7b834eda5b9063d5be9674856405255c61d WatchSource:0}: Error finding container 445b3d099e9d20d6173e2638ad2df7b834eda5b9063d5be9674856405255c61d: Status 404 returned error can't find the container with id 445b3d099e9d20d6173e2638ad2df7b834eda5b9063d5be9674856405255c61d
Sep 09 16:25:02.338903 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.338876    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"]
Sep 09 16:25:02.341548 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:25:02.341514    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b12307_d8e3_4da6_8c4a_494b3a3b0862.slice/crio-5dced465bb465175e0a87dede7a5352dcd4c242383e9626d1c60ead53d03511d WatchSource:0}: Error finding container 5dced465bb465175e0a87dede7a5352dcd4c242383e9626d1c60ead53d03511d: Status 404 returned error can't find the container with id 5dced465bb465175e0a87dede7a5352dcd4c242383e9626d1c60ead53d03511d
Sep 09 16:25:02.915293 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.915250    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj" event={"ID":"03b12307-d8e3-4da6-8c4a-494b3a3b0862","Type":"ContainerStarted","Data":"f826118c2d42efd976509107c12f815ea61a052eb5ebefc5501fa6d5bb5b46a0"}
Sep 09 16:25:02.915758 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.915313    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj" event={"ID":"03b12307-d8e3-4da6-8c4a-494b3a3b0862","Type":"ContainerStarted","Data":"5dced465bb465175e0a87dede7a5352dcd4c242383e9626d1c60ead53d03511d"}
Sep 09 16:25:02.915758 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.915345    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"
Sep 09 16:25:02.916721 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.916685    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l" event={"ID":"f55f996a-d2aa-4fc8-a38d-798a7053e0b5","Type":"ContainerStarted","Data":"9ee47ee28fd64e37c1811133418034870a940664eda4e683b13cf42311ce78e0"}
Sep 09 16:25:02.916777 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.916740    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l" event={"ID":"f55f996a-d2aa-4fc8-a38d-798a7053e0b5","Type":"ContainerStarted","Data":"445b3d099e9d20d6173e2638ad2df7b834eda5b9063d5be9674856405255c61d"}
Sep 09 16:25:02.916883 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.916866    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"
Sep 09 16:25:02.948383 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.948326    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj" podStartSLOduration=1.9483049559999999 podStartE2EDuration="1.948304956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:25:02.947825417 +0000 UTC m=+978.883443874" watchObservedRunningTime="2026-09-09 16:25:02.948304956 +0000 UTC m=+978.883923375"
Sep 09 16:25:02.986357 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:02.986299    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l" podStartSLOduration=1.9862748209999999 podStartE2EDuration="1.986274821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:25:02.984778724 +0000 UTC m=+978.920397160" watchObservedRunningTime="2026-09-09 16:25:02.986274821 +0000 UTC m=+978.921893238"
Sep 09 16:25:03.037479 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.037455    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:25:03.074157 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.074126    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7697-140146116713776_d0503b9c-dd83-4534-b2de-472b03c06371/test-curl-7697-140146116713776/0.log"
Sep 09 16:25:03.098095 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.098062    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-140146116713776"]
Sep 09 16:25:03.102237 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.102209    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-140146116713776"]
Sep 09 16:25:03.224805 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.224775    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rctwn\" (UniqueName: \"kubernetes.io/projected/d0503b9c-dd83-4534-b2de-472b03c06371-kube-api-access-rctwn\") pod \"d0503b9c-dd83-4534-b2de-472b03c06371\" (UID: \"d0503b9c-dd83-4534-b2de-472b03c06371\") "
Sep 09 16:25:03.227078 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.227055    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0503b9c-dd83-4534-b2de-472b03c06371-kube-api-access-rctwn" (OuterVolumeSpecName: "kube-api-access-rctwn") pod "d0503b9c-dd83-4534-b2de-472b03c06371" (UID: "d0503b9c-dd83-4534-b2de-472b03c06371"). InnerVolumeSpecName "kube-api-access-rctwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:25:03.231467 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.231441    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-140146116713776"]
Sep 09 16:25:03.231818 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.231800    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0503b9c-dd83-4534-b2de-472b03c06371" containerName="test-curl-7697-140146116713776"
Sep 09 16:25:03.231887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.231820    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0503b9c-dd83-4534-b2de-472b03c06371" containerName="test-curl-7697-140146116713776"
Sep 09 16:25:03.231942 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.231909    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0503b9c-dd83-4534-b2de-472b03c06371" containerName="test-curl-7697-140146116713776"
Sep 09 16:25:03.234631 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.234613    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:25:03.241541 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.241517    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-140146116713776"]
Sep 09 16:25:03.326041 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.326014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8jrc\" (UniqueName: \"kubernetes.io/projected/491866a9-2703-4b8f-8915-6ce4c15903a7-kube-api-access-v8jrc\") pod \"test-curl-7697-140146116713776\" (UID: \"491866a9-2703-4b8f-8915-6ce4c15903a7\") " pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:25:03.326157 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.326087    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rctwn\" (UniqueName: \"kubernetes.io/projected/d0503b9c-dd83-4534-b2de-472b03c06371-kube-api-access-rctwn\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:25:03.426666 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.426630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8jrc\" (UniqueName: \"kubernetes.io/projected/491866a9-2703-4b8f-8915-6ce4c15903a7-kube-api-access-v8jrc\") pod \"test-curl-7697-140146116713776\" (UID: \"491866a9-2703-4b8f-8915-6ce4c15903a7\") " pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:25:03.434322 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.434291    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8jrc\" (UniqueName: \"kubernetes.io/projected/491866a9-2703-4b8f-8915-6ce4c15903a7-kube-api-access-v8jrc\") pod \"test-curl-7697-140146116713776\" (UID: \"491866a9-2703-4b8f-8915-6ce4c15903a7\") " pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:25:03.544046 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.543941    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:25:03.812247 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.812167    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-140146116713776"]
Sep 09 16:25:03.815513 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:25:03.815484    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod491866a9_2703_4b8f_8915_6ce4c15903a7.slice/crio-51b8e6c9cf1930c0763ad6610bcfb7791750ceadd5b4a036f301cee94d48e8ec WatchSource:0}: Error finding container 51b8e6c9cf1930c0763ad6610bcfb7791750ceadd5b4a036f301cee94d48e8ec: Status 404 returned error can't find the container with id 51b8e6c9cf1930c0763ad6610bcfb7791750ceadd5b4a036f301cee94d48e8ec
Sep 09 16:25:03.922282 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.922254    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:25:03.922720 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.922263    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a3ca4f16c21b90bdf3cf6c92160cb374696b063930161d75fde2c0942969fe7"
Sep 09 16:25:03.923505 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.923473    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-140146116713776" event={"ID":"491866a9-2703-4b8f-8915-6ce4c15903a7","Type":"ContainerStarted","Data":"51b8e6c9cf1930c0763ad6610bcfb7791750ceadd5b4a036f301cee94d48e8ec"}
Sep 09 16:25:03.932841 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:03.932813    2576 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7697-140146116713776" oldPodUID="d0503b9c-dd83-4534-b2de-472b03c06371" podUID="491866a9-2703-4b8f-8915-6ce4c15903a7"
Sep 09 16:25:04.553073 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:04.553032    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0503b9c-dd83-4534-b2de-472b03c06371" path="/var/lib/kubelet/pods/d0503b9c-dd83-4534-b2de-472b03c06371/volumes"
Sep 09 16:25:05.933625 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:05.933590    2576 generic.go:356] "Generic (PLEG): container finished" podID="491866a9-2703-4b8f-8915-6ce4c15903a7" containerID="ecfd67783f29696d5434170f29408298db444e24b0035948e5e392fc6a972dee" exitCode=0
Sep 09 16:25:05.934002 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:05.933685    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-140146116713776" event={"ID":"491866a9-2703-4b8f-8915-6ce4c15903a7","Type":"ContainerDied","Data":"ecfd67783f29696d5434170f29408298db444e24b0035948e5e392fc6a972dee"}
Sep 09 16:25:07.068944 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:07.068918    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:25:07.086990 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:07.086964    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7697-140146116713776_491866a9-2703-4b8f-8915-6ce4c15903a7/test-curl-7697-140146116713776/0.log"
Sep 09 16:25:07.109508 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:07.109481    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-140146116713776"]
Sep 09 16:25:07.115380 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:07.115361    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-140146116713776"]
Sep 09 16:25:07.260504 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:07.260471    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8jrc\" (UniqueName: \"kubernetes.io/projected/491866a9-2703-4b8f-8915-6ce4c15903a7-kube-api-access-v8jrc\") pod \"491866a9-2703-4b8f-8915-6ce4c15903a7\" (UID: \"491866a9-2703-4b8f-8915-6ce4c15903a7\") "
Sep 09 16:25:07.262816 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:07.262784    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491866a9-2703-4b8f-8915-6ce4c15903a7-kube-api-access-v8jrc" (OuterVolumeSpecName: "kube-api-access-v8jrc") pod "491866a9-2703-4b8f-8915-6ce4c15903a7" (UID: "491866a9-2703-4b8f-8915-6ce4c15903a7"). InnerVolumeSpecName "kube-api-access-v8jrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:25:07.361041 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:07.361005    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8jrc\" (UniqueName: \"kubernetes.io/projected/491866a9-2703-4b8f-8915-6ce4c15903a7-kube-api-access-v8jrc\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:25:07.943830 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:07.943800    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:25:07.943830 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:07.943834    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51b8e6c9cf1930c0763ad6610bcfb7791750ceadd5b4a036f301cee94d48e8ec"
Sep 09 16:25:07.952869 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:25:07.952843    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7697-140146116713776\" is forbidden: User \"system:node:ip-10-0-137-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-117.ec2.internal' and this object" podUID="491866a9-2703-4b8f-8915-6ce4c15903a7" pod="opendatahub/test-curl-7697-140146116713776"
Sep 09 16:25:08.553077 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:08.553035    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="491866a9-2703-4b8f-8915-6ce4c15903a7" path="/var/lib/kubelet/pods/491866a9-2703-4b8f-8915-6ce4c15903a7/volumes"
Sep 09 16:25:09.188439 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.188402    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"]
Sep 09 16:25:09.188805 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.188788    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="491866a9-2703-4b8f-8915-6ce4c15903a7" containerName="test-curl-7697-140146116713776"
Sep 09 16:25:09.188888 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.188808    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="491866a9-2703-4b8f-8915-6ce4c15903a7" containerName="test-curl-7697-140146116713776"
Sep 09 16:25:09.188935 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.188904    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="491866a9-2703-4b8f-8915-6ce4c15903a7" containerName="test-curl-7697-140146116713776"
Sep 09 16:25:09.192009 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.191988    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"
Sep 09 16:25:09.194646 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.194624    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0d2e38e1-dockercfg-f28cr\""
Sep 09 16:25:09.194771 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.194650    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0d2e38e1\""
Sep 09 16:25:09.201383 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.201359    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"]
Sep 09 16:25:09.233886 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.233858    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"]
Sep 09 16:25:09.237106 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.237090    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"
Sep 09 16:25:09.248031 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.248009    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"]
Sep 09 16:25:09.376796 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.376760    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blcc8\" (UniqueName: \"kubernetes.io/projected/9c5f2538-5da5-45c3-bfa4-825227578293-kube-api-access-blcc8\") pod \"payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr\" (UID: \"9c5f2538-5da5-45c3-bfa4-825227578293\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"
Sep 09 16:25:09.376972 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.376827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd21f379-2f66-4fa9-9d40-0ae007d050dc-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr\" (UID: \"fd21f379-2f66-4fa9-9d40-0ae007d050dc\") " pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"
Sep 09 16:25:09.376972 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.376857    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5f2538-5da5-45c3-bfa4-825227578293-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr\" (UID: \"9c5f2538-5da5-45c3-bfa4-825227578293\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"
Sep 09 16:25:09.376972 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.376892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc7pt\" (UniqueName: \"kubernetes.io/projected/fd21f379-2f66-4fa9-9d40-0ae007d050dc-kube-api-access-lc7pt\") pod \"payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr\" (UID: \"fd21f379-2f66-4fa9-9d40-0ae007d050dc\") " pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"
Sep 09 16:25:09.478113 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.478082    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd21f379-2f66-4fa9-9d40-0ae007d050dc-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr\" (UID: \"fd21f379-2f66-4fa9-9d40-0ae007d050dc\") " pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"
Sep 09 16:25:09.478266 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.478123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5f2538-5da5-45c3-bfa4-825227578293-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr\" (UID: \"9c5f2538-5da5-45c3-bfa4-825227578293\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"
Sep 09 16:25:09.478266 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.478175    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc7pt\" (UniqueName: \"kubernetes.io/projected/fd21f379-2f66-4fa9-9d40-0ae007d050dc-kube-api-access-lc7pt\") pod \"payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr\" (UID: \"fd21f379-2f66-4fa9-9d40-0ae007d050dc\") " pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"
Sep 09 16:25:09.478266 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.478241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blcc8\" (UniqueName: \"kubernetes.io/projected/9c5f2538-5da5-45c3-bfa4-825227578293-kube-api-access-blcc8\") pod \"payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr\" (UID: \"9c5f2538-5da5-45c3-bfa4-825227578293\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"
Sep 09 16:25:09.478928 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.478909    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5f2538-5da5-45c3-bfa4-825227578293-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr\" (UID: \"9c5f2538-5da5-45c3-bfa4-825227578293\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"
Sep 09 16:25:09.478984 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.478922    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd21f379-2f66-4fa9-9d40-0ae007d050dc-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr\" (UID: \"fd21f379-2f66-4fa9-9d40-0ae007d050dc\") " pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"
Sep 09 16:25:09.486148 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.486121    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blcc8\" (UniqueName: \"kubernetes.io/projected/9c5f2538-5da5-45c3-bfa4-825227578293-kube-api-access-blcc8\") pod \"payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr\" (UID: \"9c5f2538-5da5-45c3-bfa4-825227578293\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"
Sep 09 16:25:09.486250 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.486223    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc7pt\" (UniqueName: \"kubernetes.io/projected/fd21f379-2f66-4fa9-9d40-0ae007d050dc-kube-api-access-lc7pt\") pod \"payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr\" (UID: \"fd21f379-2f66-4fa9-9d40-0ae007d050dc\") " pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"
Sep 09 16:25:09.502036 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.502016    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"
Sep 09 16:25:09.547125 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.547092    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"
Sep 09 16:25:09.770282 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.770146    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"]
Sep 09 16:25:09.773009 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:25:09.772981    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5f2538_5da5_45c3_bfa4_825227578293.slice/crio-f8282cf5b9f8196a34e10d51a91f63febb3c0722b17232b74273c296788ac705 WatchSource:0}: Error finding container f8282cf5b9f8196a34e10d51a91f63febb3c0722b17232b74273c296788ac705: Status 404 returned error can't find the container with id f8282cf5b9f8196a34e10d51a91f63febb3c0722b17232b74273c296788ac705
Sep 09 16:25:09.828744 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.828722    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"]
Sep 09 16:25:09.830894 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:25:09.830867    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd21f379_2f66_4fa9_9d40_0ae007d050dc.slice/crio-0238ad8fb9adb5b4f97a12d4a1257073ae23ab4572032f2f816391e11672adde WatchSource:0}: Error finding container 0238ad8fb9adb5b4f97a12d4a1257073ae23ab4572032f2f816391e11672adde: Status 404 returned error can't find the container with id 0238ad8fb9adb5b4f97a12d4a1257073ae23ab4572032f2f816391e11672adde
Sep 09 16:25:09.954649 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.954616    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr" event={"ID":"9c5f2538-5da5-45c3-bfa4-825227578293","Type":"ContainerStarted","Data":"f0a3af6f9fc07edc09d387c176a68063b2234090bc41cf91ed8e0d3f8a8ed8d7"}
Sep 09 16:25:09.954828 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.954675    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr" event={"ID":"9c5f2538-5da5-45c3-bfa4-825227578293","Type":"ContainerStarted","Data":"f8282cf5b9f8196a34e10d51a91f63febb3c0722b17232b74273c296788ac705"}
Sep 09 16:25:09.954828 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.954761    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"
Sep 09 16:25:09.956340 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.956315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr" event={"ID":"fd21f379-2f66-4fa9-9d40-0ae007d050dc","Type":"ContainerStarted","Data":"2279168cf04ef2885ac816948f5cbc6c27816976744e02cd693f89b29caa0433"}
Sep 09 16:25:09.956459 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.956359    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr" event={"ID":"fd21f379-2f66-4fa9-9d40-0ae007d050dc","Type":"ContainerStarted","Data":"0238ad8fb9adb5b4f97a12d4a1257073ae23ab4572032f2f816391e11672adde"}
Sep 09 16:25:09.956459 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.956395    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"
Sep 09 16:25:09.969858 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.969803    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr" podStartSLOduration=0.969783159 podStartE2EDuration="969.783159ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:25:09.969119594 +0000 UTC m=+985.904738021" watchObservedRunningTime="2026-09-09 16:25:09.969783159 +0000 UTC m=+985.905401575"
Sep 09 16:25:09.989198 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:09.989150    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr" podStartSLOduration=0.989133819 podStartE2EDuration="989.133819ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:25:09.9884496 +0000 UTC m=+985.924068027" watchObservedRunningTime="2026-09-09 16:25:09.989133819 +0000 UTC m=+985.924752238"
Sep 09 16:25:13.841166 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:13.841134    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"]
Sep 09 16:25:13.844377 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:13.844352    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"
Sep 09 16:25:13.847012 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:13.846992    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-743aed-dockercfg-86f8j\""
Sep 09 16:25:13.847418 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:13.847399    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-743aed\""
Sep 09 16:25:13.856186 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:13.856156    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"]
Sep 09 16:25:13.911854 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:13.911821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a40500-634d-44f6-a2d7-092442d873bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz\" (UID: \"90a40500-634d-44f6-a2d7-092442d873bb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"
Sep 09 16:25:13.912004 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:13.911875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsqzm\" (UniqueName: \"kubernetes.io/projected/90a40500-634d-44f6-a2d7-092442d873bb-kube-api-access-xsqzm\") pod \"payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz\" (UID: \"90a40500-634d-44f6-a2d7-092442d873bb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"
Sep 09 16:25:13.926072 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:13.926047    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"
Sep 09 16:25:13.926208 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:13.926108    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"
Sep 09 16:25:14.012489 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:14.012453    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a40500-634d-44f6-a2d7-092442d873bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz\" (UID: \"90a40500-634d-44f6-a2d7-092442d873bb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"
Sep 09 16:25:14.012639 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:14.012544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsqzm\" (UniqueName: \"kubernetes.io/projected/90a40500-634d-44f6-a2d7-092442d873bb-kube-api-access-xsqzm\") pod \"payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz\" (UID: \"90a40500-634d-44f6-a2d7-092442d873bb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"
Sep 09 16:25:14.013216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:14.013197    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a40500-634d-44f6-a2d7-092442d873bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz\" (UID: \"90a40500-634d-44f6-a2d7-092442d873bb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"
Sep 09 16:25:14.021030 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:14.021005    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsqzm\" (UniqueName: \"kubernetes.io/projected/90a40500-634d-44f6-a2d7-092442d873bb-kube-api-access-xsqzm\") pod \"payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz\" (UID: \"90a40500-634d-44f6-a2d7-092442d873bb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"
Sep 09 16:25:14.157558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:14.157469    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"
Sep 09 16:25:14.426086 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:14.426063    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"]
Sep 09 16:25:14.428514 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:25:14.428486    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a40500_634d_44f6_a2d7_092442d873bb.slice/crio-b3eb5ebf05202554c29fd145f42168237f7c3611e8f643488aa911a348400b1e WatchSource:0}: Error finding container b3eb5ebf05202554c29fd145f42168237f7c3611e8f643488aa911a348400b1e: Status 404 returned error can't find the container with id b3eb5ebf05202554c29fd145f42168237f7c3611e8f643488aa911a348400b1e
Sep 09 16:25:14.987161 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:14.987123    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz" event={"ID":"90a40500-634d-44f6-a2d7-092442d873bb","Type":"ContainerStarted","Data":"ea8b560e5dfbb834d14e8b291c9779720197ee9d1f3e6bb4048bda3d6956d8ec"}
Sep 09 16:25:14.987520 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:14.987187    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz" event={"ID":"90a40500-634d-44f6-a2d7-092442d873bb","Type":"ContainerStarted","Data":"b3eb5ebf05202554c29fd145f42168237f7c3611e8f643488aa911a348400b1e"}
Sep 09 16:25:14.987520 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:14.987236    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"
Sep 09 16:25:15.001839 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:15.001789    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz" podStartSLOduration=2.001774813 podStartE2EDuration="2.001774813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:25:15.000871829 +0000 UTC m=+990.936490257" watchObservedRunningTime="2026-09-09 16:25:15.001774813 +0000 UTC m=+990.937393229"
Sep 09 16:25:20.966758 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:20.966713    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"
Sep 09 16:25:20.967227 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:20.966808    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"
Sep 09 16:25:22.989605 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:22.989569    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"]
Sep 09 16:25:22.996651 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:22.996629    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:22.998848 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:22.998819    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-b90990bf-kserve-self-signed-certs\""
Sep 09 16:25:23.001930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.001884    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"]
Sep 09 16:25:23.084820 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.084779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqp9v\" (UniqueName: \"kubernetes.io/projected/85de2052-e98d-4697-8cea-d9208f88b113-kube-api-access-zqp9v\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.084820 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.084823    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-dshm\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.085092 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.084923    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-home\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.085092 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.084959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-tmp-dir\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.085092 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.084991    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-model-cache\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.085092 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.085031    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85de2052-e98d-4697-8cea-d9208f88b113-tls-certs\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.185743 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.185684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-dshm\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.185934 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.185786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-home\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.185934 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.185803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-tmp-dir\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.185934 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.185821    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-model-cache\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.185934 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.185851    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85de2052-e98d-4697-8cea-d9208f88b113-tls-certs\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.185934 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.185900    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqp9v\" (UniqueName: \"kubernetes.io/projected/85de2052-e98d-4697-8cea-d9208f88b113-kube-api-access-zqp9v\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.186249 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.186225    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-tmp-dir\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.186306 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.186234    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-model-cache\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.186360 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.186315    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-home\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.188115 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.188092    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-dshm\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.188493 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.188471    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85de2052-e98d-4697-8cea-d9208f88b113-tls-certs\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.194896 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.194869    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqp9v\" (UniqueName: \"kubernetes.io/projected/85de2052-e98d-4697-8cea-d9208f88b113-kube-api-access-zqp9v\") pod \"e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") " pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.308296 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.308207    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:23.580778 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:23.580752    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"]
Sep 09 16:25:23.583325 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:25:23.583297    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85de2052_e98d_4697_8cea_d9208f88b113.slice/crio-c50a0269bc5fb4bfc57ab3c3bd84d40a4e97abb3fbc6ba9dceace8631751f17a WatchSource:0}: Error finding container c50a0269bc5fb4bfc57ab3c3bd84d40a4e97abb3fbc6ba9dceace8631751f17a: Status 404 returned error can't find the container with id c50a0269bc5fb4bfc57ab3c3bd84d40a4e97abb3fbc6ba9dceace8631751f17a
Sep 09 16:25:24.031116 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:24.031082    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn" event={"ID":"85de2052-e98d-4697-8cea-d9208f88b113","Type":"ContainerStarted","Data":"ca865a24590c3de1968a1c548fac8fc95bab0b17cf3b1745918adeef10ada377"}
Sep 09 16:25:24.031479 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:24.031125    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn" event={"ID":"85de2052-e98d-4697-8cea-d9208f88b113","Type":"ContainerStarted","Data":"c50a0269bc5fb4bfc57ab3c3bd84d40a4e97abb3fbc6ba9dceace8631751f17a"}
Sep 09 16:25:24.047944 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:24.047891    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn" podStartSLOduration=2.047876894 podStartE2EDuration="2.047876894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:25:24.046324183 +0000 UTC m=+999.981942600" watchObservedRunningTime="2026-09-09 16:25:24.047876894 +0000 UTC m=+999.983495309"
Sep 09 16:25:25.994749 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:25.994681    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"
Sep 09 16:25:31.120936 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.120906    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"]
Sep 09 16:25:31.124466 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.124450    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"
Sep 09 16:25:31.127159 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.127134    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c23332\""
Sep 09 16:25:31.127284 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.127189    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c23332-dockercfg-xbxp7\""
Sep 09 16:25:31.138176 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.138149    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"]
Sep 09 16:25:31.148670 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.148640    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"]
Sep 09 16:25:31.152198 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.152180    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"
Sep 09 16:25:31.153006 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.152982    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cfdb\" (UniqueName: \"kubernetes.io/projected/987395c3-26da-4d13-8564-f329160dde1d-kube-api-access-9cfdb\") pod \"payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q\" (UID: \"987395c3-26da-4d13-8564-f329160dde1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"
Sep 09 16:25:31.153113 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.153019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987395c3-26da-4d13-8564-f329160dde1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q\" (UID: \"987395c3-26da-4d13-8564-f329160dde1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"
Sep 09 16:25:31.163753 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.163732    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"]
Sep 09 16:25:31.253945 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.253915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8adb5336-488c-4988-93c8-6c160ab5889d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j\" (UID: \"8adb5336-488c-4988-93c8-6c160ab5889d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"
Sep 09 16:25:31.254100 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.253978    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cfdb\" (UniqueName: \"kubernetes.io/projected/987395c3-26da-4d13-8564-f329160dde1d-kube-api-access-9cfdb\") pod \"payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q\" (UID: \"987395c3-26da-4d13-8564-f329160dde1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"
Sep 09 16:25:31.254100 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.253997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7xjk\" (UniqueName: \"kubernetes.io/projected/8adb5336-488c-4988-93c8-6c160ab5889d-kube-api-access-t7xjk\") pod \"payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j\" (UID: \"8adb5336-488c-4988-93c8-6c160ab5889d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"
Sep 09 16:25:31.254100 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.254021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987395c3-26da-4d13-8564-f329160dde1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q\" (UID: \"987395c3-26da-4d13-8564-f329160dde1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"
Sep 09 16:25:31.254670 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.254653    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987395c3-26da-4d13-8564-f329160dde1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q\" (UID: \"987395c3-26da-4d13-8564-f329160dde1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"
Sep 09 16:25:31.264747 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.264725    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cfdb\" (UniqueName: \"kubernetes.io/projected/987395c3-26da-4d13-8564-f329160dde1d-kube-api-access-9cfdb\") pod \"payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q\" (UID: \"987395c3-26da-4d13-8564-f329160dde1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"
Sep 09 16:25:31.355218 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.355169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8adb5336-488c-4988-93c8-6c160ab5889d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j\" (UID: \"8adb5336-488c-4988-93c8-6c160ab5889d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"
Sep 09 16:25:31.355399 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.355277    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7xjk\" (UniqueName: \"kubernetes.io/projected/8adb5336-488c-4988-93c8-6c160ab5889d-kube-api-access-t7xjk\") pod \"payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j\" (UID: \"8adb5336-488c-4988-93c8-6c160ab5889d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"
Sep 09 16:25:31.355919 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.355897    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8adb5336-488c-4988-93c8-6c160ab5889d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j\" (UID: \"8adb5336-488c-4988-93c8-6c160ab5889d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"
Sep 09 16:25:31.363077 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.363056    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7xjk\" (UniqueName: \"kubernetes.io/projected/8adb5336-488c-4988-93c8-6c160ab5889d-kube-api-access-t7xjk\") pod \"payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j\" (UID: \"8adb5336-488c-4988-93c8-6c160ab5889d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"
Sep 09 16:25:31.435416 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.435340    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"
Sep 09 16:25:31.465010 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.464983    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"
Sep 09 16:25:31.708464 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.708440    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"]
Sep 09 16:25:31.710639 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:25:31.710616    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987395c3_26da_4d13_8564_f329160dde1d.slice/crio-dfe915bcbfc27e690a8a535d1175a38ad6762cd9dc60b9d4682b04ea7f5a113d WatchSource:0}: Error finding container dfe915bcbfc27e690a8a535d1175a38ad6762cd9dc60b9d4682b04ea7f5a113d: Status 404 returned error can't find the container with id dfe915bcbfc27e690a8a535d1175a38ad6762cd9dc60b9d4682b04ea7f5a113d
Sep 09 16:25:31.745889 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:31.745820    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"]
Sep 09 16:25:31.752189 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:25:31.752165    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8adb5336_488c_4988_93c8_6c160ab5889d.slice/crio-be87381012442e2fc0ff9fa827b5d3755a642e75e974593a6236a38b3d5b6a25 WatchSource:0}: Error finding container be87381012442e2fc0ff9fa827b5d3755a642e75e974593a6236a38b3d5b6a25: Status 404 returned error can't find the container with id be87381012442e2fc0ff9fa827b5d3755a642e75e974593a6236a38b3d5b6a25
Sep 09 16:25:32.066983 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:32.066948    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j" event={"ID":"8adb5336-488c-4988-93c8-6c160ab5889d","Type":"ContainerStarted","Data":"58820fd27f571e87a5a2796e6d4a79ee5b5118de19118cb0b2ad9f3ab861edeb"}
Sep 09 16:25:32.067176 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:32.067006    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j" event={"ID":"8adb5336-488c-4988-93c8-6c160ab5889d","Type":"ContainerStarted","Data":"be87381012442e2fc0ff9fa827b5d3755a642e75e974593a6236a38b3d5b6a25"}
Sep 09 16:25:32.067176 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:32.067038    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"
Sep 09 16:25:32.068627 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:32.068602    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q" event={"ID":"987395c3-26da-4d13-8564-f329160dde1d","Type":"ContainerStarted","Data":"4b274b3800974c703dffd3a4ac38bb2c0751e3e079ea9fc0860e5a36d59a35bb"}
Sep 09 16:25:32.068778 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:32.068652    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q" event={"ID":"987395c3-26da-4d13-8564-f329160dde1d","Type":"ContainerStarted","Data":"dfe915bcbfc27e690a8a535d1175a38ad6762cd9dc60b9d4682b04ea7f5a113d"}
Sep 09 16:25:32.068778 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:32.068684    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"
Sep 09 16:25:32.085916 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:32.085859    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j" podStartSLOduration=1.085840079 podStartE2EDuration="1.085840079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:25:32.084128652 +0000 UTC m=+1008.019747086" watchObservedRunningTime="2026-09-09 16:25:32.085840079 +0000 UTC m=+1008.021458497"
Sep 09 16:25:32.105032 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:32.104990    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q" podStartSLOduration=1.104975856 podStartE2EDuration="1.104975856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:25:32.102627995 +0000 UTC m=+1008.038246431" watchObservedRunningTime="2026-09-09 16:25:32.104975856 +0000 UTC m=+1008.040594303"
Sep 09 16:25:33.308862 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:33.308824    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:33.309339 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:33.308879    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:33.321273 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:33.321244    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:34.089738 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:34.089710    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:25:43.076145 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:43.076113    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"
Sep 09 16:25:43.076544 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:43.076187    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"
Sep 09 16:25:49.277630 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.277596    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"]
Sep 09 16:25:49.278149 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.278100    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr" podUID="fd21f379-2f66-4fa9-9d40-0ae007d050dc" containerName="payload-processing" containerID="cri-o://2279168cf04ef2885ac816948f5cbc6c27816976744e02cd693f89b29caa0433" gracePeriod=30
Sep 09 16:25:49.288085 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.287837    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"]
Sep 09 16:25:49.288208 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.288077    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr" podUID="9c5f2538-5da5-45c3-bfa4-825227578293" containerName="payload-pre-processing" containerID="cri-o://f0a3af6f9fc07edc09d387c176a68063b2234090bc41cf91ed8e0d3f8a8ed8d7" gracePeriod=30
Sep 09 16:25:49.553029 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.553006    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"
Sep 09 16:25:49.555977 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.555958    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"
Sep 09 16:25:49.605766 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.605740    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc7pt\" (UniqueName: \"kubernetes.io/projected/fd21f379-2f66-4fa9-9d40-0ae007d050dc-kube-api-access-lc7pt\") pod \"fd21f379-2f66-4fa9-9d40-0ae007d050dc\" (UID: \"fd21f379-2f66-4fa9-9d40-0ae007d050dc\") "
Sep 09 16:25:49.605950 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.605776    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5f2538-5da5-45c3-bfa4-825227578293-plugins-config-volume\") pod \"9c5f2538-5da5-45c3-bfa4-825227578293\" (UID: \"9c5f2538-5da5-45c3-bfa4-825227578293\") "
Sep 09 16:25:49.605950 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.605838    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blcc8\" (UniqueName: \"kubernetes.io/projected/9c5f2538-5da5-45c3-bfa4-825227578293-kube-api-access-blcc8\") pod \"9c5f2538-5da5-45c3-bfa4-825227578293\" (UID: \"9c5f2538-5da5-45c3-bfa4-825227578293\") "
Sep 09 16:25:49.605950 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.605891    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd21f379-2f66-4fa9-9d40-0ae007d050dc-plugins-config-volume\") pod \"fd21f379-2f66-4fa9-9d40-0ae007d050dc\" (UID: \"fd21f379-2f66-4fa9-9d40-0ae007d050dc\") "
Sep 09 16:25:49.606420 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.606385    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5f2538-5da5-45c3-bfa4-825227578293-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c5f2538-5da5-45c3-bfa4-825227578293" (UID: "9c5f2538-5da5-45c3-bfa4-825227578293"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:25:49.606510 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.606419    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd21f379-2f66-4fa9-9d40-0ae007d050dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd21f379-2f66-4fa9-9d40-0ae007d050dc" (UID: "fd21f379-2f66-4fa9-9d40-0ae007d050dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:25:49.609435 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.609405    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5f2538-5da5-45c3-bfa4-825227578293-kube-api-access-blcc8" (OuterVolumeSpecName: "kube-api-access-blcc8") pod "9c5f2538-5da5-45c3-bfa4-825227578293" (UID: "9c5f2538-5da5-45c3-bfa4-825227578293"). InnerVolumeSpecName "kube-api-access-blcc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:25:49.609435 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.609422    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd21f379-2f66-4fa9-9d40-0ae007d050dc-kube-api-access-lc7pt" (OuterVolumeSpecName: "kube-api-access-lc7pt") pod "fd21f379-2f66-4fa9-9d40-0ae007d050dc" (UID: "fd21f379-2f66-4fa9-9d40-0ae007d050dc"). InnerVolumeSpecName "kube-api-access-lc7pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:25:49.706959 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.706926    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blcc8\" (UniqueName: \"kubernetes.io/projected/9c5f2538-5da5-45c3-bfa4-825227578293-kube-api-access-blcc8\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:25:49.706959 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.706954    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd21f379-2f66-4fa9-9d40-0ae007d050dc-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:25:49.706959 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.706962    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc7pt\" (UniqueName: \"kubernetes.io/projected/fd21f379-2f66-4fa9-9d40-0ae007d050dc-kube-api-access-lc7pt\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:25:49.707179 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:49.706972    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5f2538-5da5-45c3-bfa4-825227578293-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:25:50.154815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.154783    2576 generic.go:356] "Generic (PLEG): container finished" podID="9c5f2538-5da5-45c3-bfa4-825227578293" containerID="f0a3af6f9fc07edc09d387c176a68063b2234090bc41cf91ed8e0d3f8a8ed8d7" exitCode=0
Sep 09 16:25:50.154972 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.154827    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"
Sep 09 16:25:50.154972 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.154864    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr" event={"ID":"9c5f2538-5da5-45c3-bfa4-825227578293","Type":"ContainerDied","Data":"f0a3af6f9fc07edc09d387c176a68063b2234090bc41cf91ed8e0d3f8a8ed8d7"}
Sep 09 16:25:50.154972 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.154916    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr" event={"ID":"9c5f2538-5da5-45c3-bfa4-825227578293","Type":"ContainerDied","Data":"f8282cf5b9f8196a34e10d51a91f63febb3c0722b17232b74273c296788ac705"}
Sep 09 16:25:50.154972 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.154949    2576 scope.go:117] "RemoveContainer" containerID="f0a3af6f9fc07edc09d387c176a68063b2234090bc41cf91ed8e0d3f8a8ed8d7"
Sep 09 16:25:50.156274 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.156248    2576 generic.go:356] "Generic (PLEG): container finished" podID="fd21f379-2f66-4fa9-9d40-0ae007d050dc" containerID="2279168cf04ef2885ac816948f5cbc6c27816976744e02cd693f89b29caa0433" exitCode=0
Sep 09 16:25:50.156379 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.156295    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr" event={"ID":"fd21f379-2f66-4fa9-9d40-0ae007d050dc","Type":"ContainerDied","Data":"2279168cf04ef2885ac816948f5cbc6c27816976744e02cd693f89b29caa0433"}
Sep 09 16:25:50.156379 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.156300    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"
Sep 09 16:25:50.156379 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.156327    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr" event={"ID":"fd21f379-2f66-4fa9-9d40-0ae007d050dc","Type":"ContainerDied","Data":"0238ad8fb9adb5b4f97a12d4a1257073ae23ab4572032f2f816391e11672adde"}
Sep 09 16:25:50.165201 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.165181    2576 scope.go:117] "RemoveContainer" containerID="f0a3af6f9fc07edc09d387c176a68063b2234090bc41cf91ed8e0d3f8a8ed8d7"
Sep 09 16:25:50.165470 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:25:50.165448    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0a3af6f9fc07edc09d387c176a68063b2234090bc41cf91ed8e0d3f8a8ed8d7\": container with ID starting with f0a3af6f9fc07edc09d387c176a68063b2234090bc41cf91ed8e0d3f8a8ed8d7 not found: ID does not exist" containerID="f0a3af6f9fc07edc09d387c176a68063b2234090bc41cf91ed8e0d3f8a8ed8d7"
Sep 09 16:25:50.165544 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.165479    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a3af6f9fc07edc09d387c176a68063b2234090bc41cf91ed8e0d3f8a8ed8d7"} err="failed to get container status \"f0a3af6f9fc07edc09d387c176a68063b2234090bc41cf91ed8e0d3f8a8ed8d7\": rpc error: code = NotFound desc = could not find container \"f0a3af6f9fc07edc09d387c176a68063b2234090bc41cf91ed8e0d3f8a8ed8d7\": container with ID starting with f0a3af6f9fc07edc09d387c176a68063b2234090bc41cf91ed8e0d3f8a8ed8d7 not found: ID does not exist"
Sep 09 16:25:50.165544 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.165495    2576 scope.go:117] "RemoveContainer" containerID="2279168cf04ef2885ac816948f5cbc6c27816976744e02cd693f89b29caa0433"
Sep 09 16:25:50.175729 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.175682    2576 scope.go:117] "RemoveContainer" containerID="2279168cf04ef2885ac816948f5cbc6c27816976744e02cd693f89b29caa0433"
Sep 09 16:25:50.176017 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:25:50.175998    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2279168cf04ef2885ac816948f5cbc6c27816976744e02cd693f89b29caa0433\": container with ID starting with 2279168cf04ef2885ac816948f5cbc6c27816976744e02cd693f89b29caa0433 not found: ID does not exist" containerID="2279168cf04ef2885ac816948f5cbc6c27816976744e02cd693f89b29caa0433"
Sep 09 16:25:50.176074 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.176023    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2279168cf04ef2885ac816948f5cbc6c27816976744e02cd693f89b29caa0433"} err="failed to get container status \"2279168cf04ef2885ac816948f5cbc6c27816976744e02cd693f89b29caa0433\": rpc error: code = NotFound desc = could not find container \"2279168cf04ef2885ac816948f5cbc6c27816976744e02cd693f89b29caa0433\": container with ID starting with 2279168cf04ef2885ac816948f5cbc6c27816976744e02cd693f89b29caa0433 not found: ID does not exist"
Sep 09 16:25:50.188795 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.188772    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"]
Sep 09 16:25:50.203042 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.203019    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d2e38e1-87464bdc8-z9pbr"]
Sep 09 16:25:50.218608 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.218586    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"]
Sep 09 16:25:50.223199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.223181    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d2e38e1-7c5994d4c5-z6gmr"]
Sep 09 16:25:50.552892 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.552863    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c5f2538-5da5-45c3-bfa4-825227578293" path="/var/lib/kubelet/pods/9c5f2538-5da5-45c3-bfa4-825227578293/volumes"
Sep 09 16:25:50.553237 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:50.553209    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd21f379-2f66-4fa9-9d40-0ae007d050dc" path="/var/lib/kubelet/pods/fd21f379-2f66-4fa9-9d40-0ae007d050dc/volumes"
Sep 09 16:25:54.705107 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.705076    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"]
Sep 09 16:25:54.705538 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.705389    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd21f379-2f66-4fa9-9d40-0ae007d050dc" containerName="payload-processing"
Sep 09 16:25:54.705538 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.705400    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd21f379-2f66-4fa9-9d40-0ae007d050dc" containerName="payload-processing"
Sep 09 16:25:54.705538 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.705413    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c5f2538-5da5-45c3-bfa4-825227578293" containerName="payload-pre-processing"
Sep 09 16:25:54.705538 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.705419    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c5f2538-5da5-45c3-bfa4-825227578293" containerName="payload-pre-processing"
Sep 09 16:25:54.705538 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.705469    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c5f2538-5da5-45c3-bfa4-825227578293" containerName="payload-pre-processing"
Sep 09 16:25:54.705538 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.705479    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd21f379-2f66-4fa9-9d40-0ae007d050dc" containerName="payload-processing"
Sep 09 16:25:54.709986 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.709968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"
Sep 09 16:25:54.712314 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.712288    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fb79f974\""
Sep 09 16:25:54.712447 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.712384    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fb79f974-dockercfg-r24rw\""
Sep 09 16:25:54.723075 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.723053    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"]
Sep 09 16:25:54.752238 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.752209    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"]
Sep 09 16:25:54.755281 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.755266    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"
Sep 09 16:25:54.766057 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.766033    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"]
Sep 09 16:25:54.851454 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.851423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ksv\" (UniqueName: \"kubernetes.io/projected/a2ca3cab-eddf-48a3-946c-2b1de2b43d0d-kube-api-access-b4ksv\") pod \"payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k\" (UID: \"a2ca3cab-eddf-48a3-946c-2b1de2b43d0d\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"
Sep 09 16:25:54.851622 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.851492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63646774-3602-4908-9111-8cae156701d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6\" (UID: \"63646774-3602-4908-9111-8cae156701d7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"
Sep 09 16:25:54.851622 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.851560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ca3cab-eddf-48a3-946c-2b1de2b43d0d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k\" (UID: \"a2ca3cab-eddf-48a3-946c-2b1de2b43d0d\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"
Sep 09 16:25:54.851622 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.851584    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxnd\" (UniqueName: \"kubernetes.io/projected/63646774-3602-4908-9111-8cae156701d7-kube-api-access-5kxnd\") pod \"payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6\" (UID: \"63646774-3602-4908-9111-8cae156701d7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"
Sep 09 16:25:54.952453 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.952414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ksv\" (UniqueName: \"kubernetes.io/projected/a2ca3cab-eddf-48a3-946c-2b1de2b43d0d-kube-api-access-b4ksv\") pod \"payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k\" (UID: \"a2ca3cab-eddf-48a3-946c-2b1de2b43d0d\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"
Sep 09 16:25:54.952627 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.952490    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63646774-3602-4908-9111-8cae156701d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6\" (UID: \"63646774-3602-4908-9111-8cae156701d7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"
Sep 09 16:25:54.952627 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.952523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ca3cab-eddf-48a3-946c-2b1de2b43d0d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k\" (UID: \"a2ca3cab-eddf-48a3-946c-2b1de2b43d0d\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"
Sep 09 16:25:54.952627 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.952540    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxnd\" (UniqueName: \"kubernetes.io/projected/63646774-3602-4908-9111-8cae156701d7-kube-api-access-5kxnd\") pod \"payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6\" (UID: \"63646774-3602-4908-9111-8cae156701d7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"
Sep 09 16:25:54.953274 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.953252    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ca3cab-eddf-48a3-946c-2b1de2b43d0d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k\" (UID: \"a2ca3cab-eddf-48a3-946c-2b1de2b43d0d\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"
Sep 09 16:25:54.953314 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.953260    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63646774-3602-4908-9111-8cae156701d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6\" (UID: \"63646774-3602-4908-9111-8cae156701d7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"
Sep 09 16:25:54.960861 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.960795    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ksv\" (UniqueName: \"kubernetes.io/projected/a2ca3cab-eddf-48a3-946c-2b1de2b43d0d-kube-api-access-b4ksv\") pod \"payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k\" (UID: \"a2ca3cab-eddf-48a3-946c-2b1de2b43d0d\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"
Sep 09 16:25:54.960861 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:54.960815    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxnd\" (UniqueName: \"kubernetes.io/projected/63646774-3602-4908-9111-8cae156701d7-kube-api-access-5kxnd\") pod \"payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6\" (UID: \"63646774-3602-4908-9111-8cae156701d7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"
Sep 09 16:25:55.020949 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:55.020918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"
Sep 09 16:25:55.065202 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:55.065164    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"
Sep 09 16:25:55.294271 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:55.294232    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"]
Sep 09 16:25:55.296426 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:25:55.296401    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63646774_3602_4908_9111_8cae156701d7.slice/crio-6c27a9ec6bca64b1455c7f0eb76b25d6151e44e1b7285ca96ee8cc3ccfcccbb3 WatchSource:0}: Error finding container 6c27a9ec6bca64b1455c7f0eb76b25d6151e44e1b7285ca96ee8cc3ccfcccbb3: Status 404 returned error can't find the container with id 6c27a9ec6bca64b1455c7f0eb76b25d6151e44e1b7285ca96ee8cc3ccfcccbb3
Sep 09 16:25:55.345498 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:55.345427    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"]
Sep 09 16:25:55.347546 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:25:55.347522    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ca3cab_eddf_48a3_946c_2b1de2b43d0d.slice/crio-3835013c85ddb4c90f7da9d06a29dbd5d4b2fce27039fcb812b17ed948e11c83 WatchSource:0}: Error finding container 3835013c85ddb4c90f7da9d06a29dbd5d4b2fce27039fcb812b17ed948e11c83: Status 404 returned error can't find the container with id 3835013c85ddb4c90f7da9d06a29dbd5d4b2fce27039fcb812b17ed948e11c83
Sep 09 16:25:56.187050 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:56.187006    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k" event={"ID":"a2ca3cab-eddf-48a3-946c-2b1de2b43d0d","Type":"ContainerStarted","Data":"0931ece2816f07c38ed4c52df9770cdc07b772a9aa5f8d3f4b8cbb8a37fd71b2"}
Sep 09 16:25:56.187558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:56.187071    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k" event={"ID":"a2ca3cab-eddf-48a3-946c-2b1de2b43d0d","Type":"ContainerStarted","Data":"3835013c85ddb4c90f7da9d06a29dbd5d4b2fce27039fcb812b17ed948e11c83"}
Sep 09 16:25:56.187558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:56.187102    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"
Sep 09 16:25:56.188549 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:56.188525    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6" event={"ID":"63646774-3602-4908-9111-8cae156701d7","Type":"ContainerStarted","Data":"894c944d36ca19d9a9fcb2e79c8b9f4732489f7c01241f2d7afb29c402e76726"}
Sep 09 16:25:56.188612 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:56.188574    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6" event={"ID":"63646774-3602-4908-9111-8cae156701d7","Type":"ContainerStarted","Data":"6c27a9ec6bca64b1455c7f0eb76b25d6151e44e1b7285ca96ee8cc3ccfcccbb3"}
Sep 09 16:25:56.188669 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:56.188612    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"
Sep 09 16:25:56.204999 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:56.204955    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k" podStartSLOduration=2.204940743 podStartE2EDuration="2.204940743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:25:56.203638756 +0000 UTC m=+1032.139257196" watchObservedRunningTime="2026-09-09 16:25:56.204940743 +0000 UTC m=+1032.140559159"
Sep 09 16:25:56.223206 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:25:56.223164    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6" podStartSLOduration=2.223153489 podStartE2EDuration="2.223153489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:25:56.222904845 +0000 UTC m=+1032.158523265" watchObservedRunningTime="2026-09-09 16:25:56.223153489 +0000 UTC m=+1032.158771904"
Sep 09 16:26:07.195839 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:07.195809    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"
Sep 09 16:26:07.196262 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:07.195891    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"
Sep 09 16:26:19.694323 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.694285    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"]
Sep 09 16:26:19.697810 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.697786    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.700068 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.700044    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-b90990bf-kserve-self-signed-certs\""
Sep 09 16:26:19.708830 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.708809    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"]
Sep 09 16:26:19.765355 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.765320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-dshm\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.765537 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.765363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-model-cache\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.765537 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.765406    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-home\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.765537 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.765427    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-tmp-dir\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.765537 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.765462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptdmj\" (UniqueName: \"kubernetes.io/projected/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-kube-api-access-ptdmj\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.765537 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.765480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-tls-certs\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.865812 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.865782    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-tmp-dir\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.865985 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.865829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptdmj\" (UniqueName: \"kubernetes.io/projected/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-kube-api-access-ptdmj\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.865985 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.865849    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-tls-certs\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.865985 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.865899    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-dshm\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.865985 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.865920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-model-cache\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.865985 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.865941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-home\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.866295 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.866266    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-tmp-dir\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.866363 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.866285    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-home\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.866398 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.866356    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-model-cache\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.868457 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.868435    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-dshm\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.868568 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.868498    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-tls-certs\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:19.873438 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:19.873414    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptdmj\" (UniqueName: \"kubernetes.io/projected/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-kube-api-access-ptdmj\") pod \"e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") " pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:20.007079 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:20.007042    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:20.277003 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:20.276883    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"]
Sep 09 16:26:20.279239 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:26:20.279206    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e0a2e41_53ac_463f_99f8_4c12f991d0a2.slice/crio-ac4f42219e011de6455268136864540146b39c7b0330da3cee348e7bb7628a0b WatchSource:0}: Error finding container ac4f42219e011de6455268136864540146b39c7b0330da3cee348e7bb7628a0b: Status 404 returned error can't find the container with id ac4f42219e011de6455268136864540146b39c7b0330da3cee348e7bb7628a0b
Sep 09 16:26:20.301590 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:20.301567    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td" event={"ID":"7e0a2e41-53ac-463f-99f8-4c12f991d0a2","Type":"ContainerStarted","Data":"ac4f42219e011de6455268136864540146b39c7b0330da3cee348e7bb7628a0b"}
Sep 09 16:26:21.308059 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:21.308020    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td" event={"ID":"7e0a2e41-53ac-463f-99f8-4c12f991d0a2","Type":"ContainerStarted","Data":"c11d6df26429ebfcda2cdfffb2a3d37942dff95a182f7ceb055e7a04ae54ece9"}
Sep 09 16:26:21.323823 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:21.323772    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td" podStartSLOduration=2.323756137 podStartE2EDuration="2.323756137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:26:21.323169428 +0000 UTC m=+1057.258787854" watchObservedRunningTime="2026-09-09 16:26:21.323756137 +0000 UTC m=+1057.259374553"
Sep 09 16:26:25.358968 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.358929    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"]
Sep 09 16:26:25.359443 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.359184    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k" podUID="a2ca3cab-eddf-48a3-946c-2b1de2b43d0d" containerName="payload-processing" containerID="cri-o://0931ece2816f07c38ed4c52df9770cdc07b772a9aa5f8d3f4b8cbb8a37fd71b2" gracePeriod=30
Sep 09 16:26:25.364098 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.364070    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"]
Sep 09 16:26:25.364481 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.364456    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6" podUID="63646774-3602-4908-9111-8cae156701d7" containerName="payload-pre-processing" containerID="cri-o://894c944d36ca19d9a9fcb2e79c8b9f4732489f7c01241f2d7afb29c402e76726" gracePeriod=30
Sep 09 16:26:25.665336 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.665244    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"
Sep 09 16:26:25.705171 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.705139    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"
Sep 09 16:26:25.718461 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.718426    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63646774-3602-4908-9111-8cae156701d7-plugins-config-volume\") pod \"63646774-3602-4908-9111-8cae156701d7\" (UID: \"63646774-3602-4908-9111-8cae156701d7\") "
Sep 09 16:26:25.718461 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.718463    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kxnd\" (UniqueName: \"kubernetes.io/projected/63646774-3602-4908-9111-8cae156701d7-kube-api-access-5kxnd\") pod \"63646774-3602-4908-9111-8cae156701d7\" (UID: \"63646774-3602-4908-9111-8cae156701d7\") "
Sep 09 16:26:25.718842 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.718489    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ca3cab-eddf-48a3-946c-2b1de2b43d0d-plugins-config-volume\") pod \"a2ca3cab-eddf-48a3-946c-2b1de2b43d0d\" (UID: \"a2ca3cab-eddf-48a3-946c-2b1de2b43d0d\") "
Sep 09 16:26:25.718842 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.718576    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4ksv\" (UniqueName: \"kubernetes.io/projected/a2ca3cab-eddf-48a3-946c-2b1de2b43d0d-kube-api-access-b4ksv\") pod \"a2ca3cab-eddf-48a3-946c-2b1de2b43d0d\" (UID: \"a2ca3cab-eddf-48a3-946c-2b1de2b43d0d\") "
Sep 09 16:26:25.718955 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.718897    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ca3cab-eddf-48a3-946c-2b1de2b43d0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2ca3cab-eddf-48a3-946c-2b1de2b43d0d" (UID: "a2ca3cab-eddf-48a3-946c-2b1de2b43d0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:26:25.719013 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.718902    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63646774-3602-4908-9111-8cae156701d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63646774-3602-4908-9111-8cae156701d7" (UID: "63646774-3602-4908-9111-8cae156701d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:26:25.719137 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.719115    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63646774-3602-4908-9111-8cae156701d7-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:26:25.719204 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.719142    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ca3cab-eddf-48a3-946c-2b1de2b43d0d-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:26:25.721240 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.721209    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63646774-3602-4908-9111-8cae156701d7-kube-api-access-5kxnd" (OuterVolumeSpecName: "kube-api-access-5kxnd") pod "63646774-3602-4908-9111-8cae156701d7" (UID: "63646774-3602-4908-9111-8cae156701d7"). InnerVolumeSpecName "kube-api-access-5kxnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:26:25.722655 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.722622    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ca3cab-eddf-48a3-946c-2b1de2b43d0d-kube-api-access-b4ksv" (OuterVolumeSpecName: "kube-api-access-b4ksv") pod "a2ca3cab-eddf-48a3-946c-2b1de2b43d0d" (UID: "a2ca3cab-eddf-48a3-946c-2b1de2b43d0d"). InnerVolumeSpecName "kube-api-access-b4ksv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:26:25.820066 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.820029    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4ksv\" (UniqueName: \"kubernetes.io/projected/a2ca3cab-eddf-48a3-946c-2b1de2b43d0d-kube-api-access-b4ksv\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:26:25.820066 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:25.820058    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kxnd\" (UniqueName: \"kubernetes.io/projected/63646774-3602-4908-9111-8cae156701d7-kube-api-access-5kxnd\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:26:26.333846 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.333814    2576 generic.go:356] "Generic (PLEG): container finished" podID="63646774-3602-4908-9111-8cae156701d7" containerID="894c944d36ca19d9a9fcb2e79c8b9f4732489f7c01241f2d7afb29c402e76726" exitCode=0
Sep 09 16:26:26.334048 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.333856    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"
Sep 09 16:26:26.334048 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.333900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6" event={"ID":"63646774-3602-4908-9111-8cae156701d7","Type":"ContainerDied","Data":"894c944d36ca19d9a9fcb2e79c8b9f4732489f7c01241f2d7afb29c402e76726"}
Sep 09 16:26:26.334048 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.333956    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6" event={"ID":"63646774-3602-4908-9111-8cae156701d7","Type":"ContainerDied","Data":"6c27a9ec6bca64b1455c7f0eb76b25d6151e44e1b7285ca96ee8cc3ccfcccbb3"}
Sep 09 16:26:26.334048 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.333988    2576 scope.go:117] "RemoveContainer" containerID="894c944d36ca19d9a9fcb2e79c8b9f4732489f7c01241f2d7afb29c402e76726"
Sep 09 16:26:26.341659 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.341463    2576 generic.go:356] "Generic (PLEG): container finished" podID="a2ca3cab-eddf-48a3-946c-2b1de2b43d0d" containerID="0931ece2816f07c38ed4c52df9770cdc07b772a9aa5f8d3f4b8cbb8a37fd71b2" exitCode=0
Sep 09 16:26:26.341814 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.341635    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k" event={"ID":"a2ca3cab-eddf-48a3-946c-2b1de2b43d0d","Type":"ContainerDied","Data":"0931ece2816f07c38ed4c52df9770cdc07b772a9aa5f8d3f4b8cbb8a37fd71b2"}
Sep 09 16:26:26.341814 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.341721    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"
Sep 09 16:26:26.341814 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.341781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k" event={"ID":"a2ca3cab-eddf-48a3-946c-2b1de2b43d0d","Type":"ContainerDied","Data":"3835013c85ddb4c90f7da9d06a29dbd5d4b2fce27039fcb812b17ed948e11c83"}
Sep 09 16:26:26.350184 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.350102    2576 scope.go:117] "RemoveContainer" containerID="894c944d36ca19d9a9fcb2e79c8b9f4732489f7c01241f2d7afb29c402e76726"
Sep 09 16:26:26.350528 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:26:26.350508    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894c944d36ca19d9a9fcb2e79c8b9f4732489f7c01241f2d7afb29c402e76726\": container with ID starting with 894c944d36ca19d9a9fcb2e79c8b9f4732489f7c01241f2d7afb29c402e76726 not found: ID does not exist" containerID="894c944d36ca19d9a9fcb2e79c8b9f4732489f7c01241f2d7afb29c402e76726"
Sep 09 16:26:26.350598 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.350537    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894c944d36ca19d9a9fcb2e79c8b9f4732489f7c01241f2d7afb29c402e76726"} err="failed to get container status \"894c944d36ca19d9a9fcb2e79c8b9f4732489f7c01241f2d7afb29c402e76726\": rpc error: code = NotFound desc = could not find container \"894c944d36ca19d9a9fcb2e79c8b9f4732489f7c01241f2d7afb29c402e76726\": container with ID starting with 894c944d36ca19d9a9fcb2e79c8b9f4732489f7c01241f2d7afb29c402e76726 not found: ID does not exist"
Sep 09 16:26:26.350598 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.350559    2576 scope.go:117] "RemoveContainer" containerID="0931ece2816f07c38ed4c52df9770cdc07b772a9aa5f8d3f4b8cbb8a37fd71b2"
Sep 09 16:26:26.358687 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.358667    2576 scope.go:117] "RemoveContainer" containerID="0931ece2816f07c38ed4c52df9770cdc07b772a9aa5f8d3f4b8cbb8a37fd71b2"
Sep 09 16:26:26.358964 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.358938    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"]
Sep 09 16:26:26.358964 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:26:26.358945    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0931ece2816f07c38ed4c52df9770cdc07b772a9aa5f8d3f4b8cbb8a37fd71b2\": container with ID starting with 0931ece2816f07c38ed4c52df9770cdc07b772a9aa5f8d3f4b8cbb8a37fd71b2 not found: ID does not exist" containerID="0931ece2816f07c38ed4c52df9770cdc07b772a9aa5f8d3f4b8cbb8a37fd71b2"
Sep 09 16:26:26.359250 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.358974    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0931ece2816f07c38ed4c52df9770cdc07b772a9aa5f8d3f4b8cbb8a37fd71b2"} err="failed to get container status \"0931ece2816f07c38ed4c52df9770cdc07b772a9aa5f8d3f4b8cbb8a37fd71b2\": rpc error: code = NotFound desc = could not find container \"0931ece2816f07c38ed4c52df9770cdc07b772a9aa5f8d3f4b8cbb8a37fd71b2\": container with ID starting with 0931ece2816f07c38ed4c52df9770cdc07b772a9aa5f8d3f4b8cbb8a37fd71b2 not found: ID does not exist"
Sep 09 16:26:26.363036 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.363016    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb79f974-c6d8f8fb6-k8hb6"]
Sep 09 16:26:26.377354 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.377322    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"]
Sep 09 16:26:26.381671 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.381652    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb79f974-795b75c594-fmh2k"]
Sep 09 16:26:26.553562 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.553528    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63646774-3602-4908-9111-8cae156701d7" path="/var/lib/kubelet/pods/63646774-3602-4908-9111-8cae156701d7/volumes"
Sep 09 16:26:26.553899 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:26.553884    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2ca3cab-eddf-48a3-946c-2b1de2b43d0d" path="/var/lib/kubelet/pods/a2ca3cab-eddf-48a3-946c-2b1de2b43d0d/volumes"
Sep 09 16:26:30.008199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.008163    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:30.008199 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.008204    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:30.020718 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.020669    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:30.372413 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.372318    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:26:30.591001 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.590968    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"]
Sep 09 16:26:30.591317 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.591304    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63646774-3602-4908-9111-8cae156701d7" containerName="payload-pre-processing"
Sep 09 16:26:30.591358 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.591319    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63646774-3602-4908-9111-8cae156701d7" containerName="payload-pre-processing"
Sep 09 16:26:30.591358 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.591338    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ca3cab-eddf-48a3-946c-2b1de2b43d0d" containerName="payload-processing"
Sep 09 16:26:30.591358 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.591348    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ca3cab-eddf-48a3-946c-2b1de2b43d0d" containerName="payload-processing"
Sep 09 16:26:30.591454 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.591404    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="63646774-3602-4908-9111-8cae156701d7" containerName="payload-pre-processing"
Sep 09 16:26:30.591454 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.591412    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ca3cab-eddf-48a3-946c-2b1de2b43d0d" containerName="payload-processing"
Sep 09 16:26:30.595924 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.595907    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"
Sep 09 16:26:30.598319 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.598302    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8a375fa0\""
Sep 09 16:26:30.605806 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.605786    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"]
Sep 09 16:26:30.664367 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.664281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv7wc\" (UniqueName: \"kubernetes.io/projected/2b603c64-fbd6-4401-8881-c5e65c46794b-kube-api-access-rv7wc\") pod \"payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2\" (UID: \"2b603c64-fbd6-4401-8881-c5e65c46794b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"
Sep 09 16:26:30.664367 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.664362    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b603c64-fbd6-4401-8881-c5e65c46794b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2\" (UID: \"2b603c64-fbd6-4401-8881-c5e65c46794b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"
Sep 09 16:26:30.765658 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.765608    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv7wc\" (UniqueName: \"kubernetes.io/projected/2b603c64-fbd6-4401-8881-c5e65c46794b-kube-api-access-rv7wc\") pod \"payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2\" (UID: \"2b603c64-fbd6-4401-8881-c5e65c46794b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"
Sep 09 16:26:30.765658 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.765665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b603c64-fbd6-4401-8881-c5e65c46794b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2\" (UID: \"2b603c64-fbd6-4401-8881-c5e65c46794b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"
Sep 09 16:26:30.766390 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.766363    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b603c64-fbd6-4401-8881-c5e65c46794b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2\" (UID: \"2b603c64-fbd6-4401-8881-c5e65c46794b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"
Sep 09 16:26:30.773413 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.773382    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv7wc\" (UniqueName: \"kubernetes.io/projected/2b603c64-fbd6-4401-8881-c5e65c46794b-kube-api-access-rv7wc\") pod \"payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2\" (UID: \"2b603c64-fbd6-4401-8881-c5e65c46794b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"
Sep 09 16:26:30.906810 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:30.906775    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"
Sep 09 16:26:31.179520 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:31.179492    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"]
Sep 09 16:26:31.181742 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:26:31.181691    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b603c64_fbd6_4401_8881_c5e65c46794b.slice/crio-2e20a6cf03c41e1fc6588bd6792ce9d6f19ae217f5f43ddd44caf68c25cc2ee6 WatchSource:0}: Error finding container 2e20a6cf03c41e1fc6588bd6792ce9d6f19ae217f5f43ddd44caf68c25cc2ee6: Status 404 returned error can't find the container with id 2e20a6cf03c41e1fc6588bd6792ce9d6f19ae217f5f43ddd44caf68c25cc2ee6
Sep 09 16:26:31.366938 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:31.366902    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2" event={"ID":"2b603c64-fbd6-4401-8881-c5e65c46794b","Type":"ContainerStarted","Data":"b086b69771b0bf8dc8182f3786c19817153265837692d527b0a1413554336074"}
Sep 09 16:26:31.367110 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:31.366950    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2" event={"ID":"2b603c64-fbd6-4401-8881-c5e65c46794b","Type":"ContainerStarted","Data":"2e20a6cf03c41e1fc6588bd6792ce9d6f19ae217f5f43ddd44caf68c25cc2ee6"}
Sep 09 16:26:31.367110 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:31.367016    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"
Sep 09 16:26:31.383022 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:31.382971    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2" podStartSLOduration=1.382957091 podStartE2EDuration="1.382957091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:26:31.381116552 +0000 UTC m=+1067.316734989" watchObservedRunningTime="2026-09-09 16:26:31.382957091 +0000 UTC m=+1067.318575506"
Sep 09 16:26:32.788441 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.788411    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"]
Sep 09 16:26:32.791682 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.791665    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.794880 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.794859    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-743aed\"/\"autores-model-743aed-kserve-self-signed-certs\""
Sep 09 16:26:32.794880 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.794868    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-743aed\"/\"openshift-service-ca.crt\""
Sep 09 16:26:32.795064 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.794921    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-743aed\"/\"kube-root-ca.crt\""
Sep 09 16:26:32.795064 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.794873    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-743aed\"/\"default-dockercfg-wlm8c\""
Sep 09 16:26:32.801871 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.801850    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"]
Sep 09 16:26:32.884254 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.884231    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d09e129-d324-43d6-ab55-e3c974ac178e-tls-certs\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.884402 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.884283    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-dshm\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.884402 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.884339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-home\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.884402 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.884361    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-tmp-dir\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.884402 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.884381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhx7r\" (UniqueName: \"kubernetes.io/projected/1d09e129-d324-43d6-ab55-e3c974ac178e-kube-api-access-zhx7r\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.884553 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.884426    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-model-cache\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.985046 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.985019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-dshm\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.985189 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.985084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-home\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.985189 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.985117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-tmp-dir\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.985189 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.985141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhx7r\" (UniqueName: \"kubernetes.io/projected/1d09e129-d324-43d6-ab55-e3c974ac178e-kube-api-access-zhx7r\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.985189 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.985187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-model-cache\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.985386 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.985226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d09e129-d324-43d6-ab55-e3c974ac178e-tls-certs\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.985586 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.985562    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-home\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.985586 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.985576    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-tmp-dir\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.985745 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.985653    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-model-cache\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.987502 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.987484    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-dshm\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.987889 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.987869    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d09e129-d324-43d6-ab55-e3c974ac178e-tls-certs\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:32.992455 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:32.992435    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhx7r\" (UniqueName: \"kubernetes.io/projected/1d09e129-d324-43d6-ab55-e3c974ac178e-kube-api-access-zhx7r\") pod \"autores-model-743aed-kserve-577bc6d87f-4hc4d\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") " pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:33.100676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:33.100602    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:33.368825 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:33.368797    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"]
Sep 09 16:26:33.371303 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:26:33.371276    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d09e129_d324_43d6_ab55_e3c974ac178e.slice/crio-340a26292bcfb85cf09e9e312732c0cb44a875b45901f88798fdcc06156ba5f7 WatchSource:0}: Error finding container 340a26292bcfb85cf09e9e312732c0cb44a875b45901f88798fdcc06156ba5f7: Status 404 returned error can't find the container with id 340a26292bcfb85cf09e9e312732c0cb44a875b45901f88798fdcc06156ba5f7
Sep 09 16:26:33.377558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:33.377531    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d" event={"ID":"1d09e129-d324-43d6-ab55-e3c974ac178e","Type":"ContainerStarted","Data":"340a26292bcfb85cf09e9e312732c0cb44a875b45901f88798fdcc06156ba5f7"}
Sep 09 16:26:34.383753 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:34.383712    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d" event={"ID":"1d09e129-d324-43d6-ab55-e3c974ac178e","Type":"ContainerStarted","Data":"574430fd3b5aa166c679f10f0049ce82614ec4dffdb250d8691b8502b534faab"}
Sep 09 16:26:34.399387 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:34.399340    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d" podStartSLOduration=2.399326644 podStartE2EDuration="2.399326644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:26:34.398198117 +0000 UTC m=+1070.333816552" watchObservedRunningTime="2026-09-09 16:26:34.399326644 +0000 UTC m=+1070.334945059"
Sep 09 16:26:42.373209 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:42.373132    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"
Sep 09 16:26:43.100809 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:43.100775    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:43.100974 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:43.100822    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:43.113718 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:43.113673    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:43.434876 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:43.434803    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:26:49.643387 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:49.643358    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"]
Sep 09 16:26:50.290908 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:26:50.290882    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"]
Sep 09 16:27:00.603769 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:00.603730    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"]
Sep 09 16:27:00.604329 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:00.603948    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz" podUID="90a40500-634d-44f6-a2d7-092442d873bb" containerName="payload-pre-processing" containerID="cri-o://ea8b560e5dfbb834d14e8b291c9779720197ee9d1f3e6bb4048bda3d6956d8ec" gracePeriod=30
Sep 09 16:27:00.745829 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:00.745795    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"]
Sep 09 16:27:00.746251 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:00.746209    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2" podUID="2b603c64-fbd6-4401-8881-c5e65c46794b" containerName="payload-pre-processing" containerID="cri-o://b086b69771b0bf8dc8182f3786c19817153265837692d527b0a1413554336074" gracePeriod=30
Sep 09 16:27:00.872137 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:00.872085    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"
Sep 09 16:27:00.987184 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:00.987160    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"
Sep 09 16:27:01.030800 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.030749    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a40500-634d-44f6-a2d7-092442d873bb-plugins-config-volume\") pod \"90a40500-634d-44f6-a2d7-092442d873bb\" (UID: \"90a40500-634d-44f6-a2d7-092442d873bb\") "
Sep 09 16:27:01.030962 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.030879    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsqzm\" (UniqueName: \"kubernetes.io/projected/90a40500-634d-44f6-a2d7-092442d873bb-kube-api-access-xsqzm\") pod \"90a40500-634d-44f6-a2d7-092442d873bb\" (UID: \"90a40500-634d-44f6-a2d7-092442d873bb\") "
Sep 09 16:27:01.031112 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.031090    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a40500-634d-44f6-a2d7-092442d873bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90a40500-634d-44f6-a2d7-092442d873bb" (UID: "90a40500-634d-44f6-a2d7-092442d873bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:27:01.031189 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.031172    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a40500-634d-44f6-a2d7-092442d873bb-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:01.033066 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.033036    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a40500-634d-44f6-a2d7-092442d873bb-kube-api-access-xsqzm" (OuterVolumeSpecName: "kube-api-access-xsqzm") pod "90a40500-634d-44f6-a2d7-092442d873bb" (UID: "90a40500-634d-44f6-a2d7-092442d873bb"). InnerVolumeSpecName "kube-api-access-xsqzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:27:01.132230 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.132137    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv7wc\" (UniqueName: \"kubernetes.io/projected/2b603c64-fbd6-4401-8881-c5e65c46794b-kube-api-access-rv7wc\") pod \"2b603c64-fbd6-4401-8881-c5e65c46794b\" (UID: \"2b603c64-fbd6-4401-8881-c5e65c46794b\") "
Sep 09 16:27:01.132402 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.132236    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b603c64-fbd6-4401-8881-c5e65c46794b-plugins-config-volume\") pod \"2b603c64-fbd6-4401-8881-c5e65c46794b\" (UID: \"2b603c64-fbd6-4401-8881-c5e65c46794b\") "
Sep 09 16:27:01.132481 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.132465    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsqzm\" (UniqueName: \"kubernetes.io/projected/90a40500-634d-44f6-a2d7-092442d873bb-kube-api-access-xsqzm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:01.132578 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.132556    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b603c64-fbd6-4401-8881-c5e65c46794b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b603c64-fbd6-4401-8881-c5e65c46794b" (UID: "2b603c64-fbd6-4401-8881-c5e65c46794b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:27:01.134346 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.134327    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b603c64-fbd6-4401-8881-c5e65c46794b-kube-api-access-rv7wc" (OuterVolumeSpecName: "kube-api-access-rv7wc") pod "2b603c64-fbd6-4401-8881-c5e65c46794b" (UID: "2b603c64-fbd6-4401-8881-c5e65c46794b"). InnerVolumeSpecName "kube-api-access-rv7wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:27:01.233577 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.233543    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv7wc\" (UniqueName: \"kubernetes.io/projected/2b603c64-fbd6-4401-8881-c5e65c46794b-kube-api-access-rv7wc\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:01.233577 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.233570    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b603c64-fbd6-4401-8881-c5e65c46794b-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:01.506127 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.506089    2576 generic.go:356] "Generic (PLEG): container finished" podID="2b603c64-fbd6-4401-8881-c5e65c46794b" containerID="b086b69771b0bf8dc8182f3786c19817153265837692d527b0a1413554336074" exitCode=0
Sep 09 16:27:01.506334 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.506143    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"
Sep 09 16:27:01.506334 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.506186    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2" event={"ID":"2b603c64-fbd6-4401-8881-c5e65c46794b","Type":"ContainerDied","Data":"b086b69771b0bf8dc8182f3786c19817153265837692d527b0a1413554336074"}
Sep 09 16:27:01.506334 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.506274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2" event={"ID":"2b603c64-fbd6-4401-8881-c5e65c46794b","Type":"ContainerDied","Data":"2e20a6cf03c41e1fc6588bd6792ce9d6f19ae217f5f43ddd44caf68c25cc2ee6"}
Sep 09 16:27:01.506334 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.506301    2576 scope.go:117] "RemoveContainer" containerID="b086b69771b0bf8dc8182f3786c19817153265837692d527b0a1413554336074"
Sep 09 16:27:01.507689 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.507671    2576 generic.go:356] "Generic (PLEG): container finished" podID="90a40500-634d-44f6-a2d7-092442d873bb" containerID="ea8b560e5dfbb834d14e8b291c9779720197ee9d1f3e6bb4048bda3d6956d8ec" exitCode=0
Sep 09 16:27:01.507787 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.507727    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz" event={"ID":"90a40500-634d-44f6-a2d7-092442d873bb","Type":"ContainerDied","Data":"ea8b560e5dfbb834d14e8b291c9779720197ee9d1f3e6bb4048bda3d6956d8ec"}
Sep 09 16:27:01.507787 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.507755    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"
Sep 09 16:27:01.507899 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.507760    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz" event={"ID":"90a40500-634d-44f6-a2d7-092442d873bb","Type":"ContainerDied","Data":"b3eb5ebf05202554c29fd145f42168237f7c3611e8f643488aa911a348400b1e"}
Sep 09 16:27:01.514860 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.514842    2576 scope.go:117] "RemoveContainer" containerID="b086b69771b0bf8dc8182f3786c19817153265837692d527b0a1413554336074"
Sep 09 16:27:01.515135 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:27:01.515112    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b086b69771b0bf8dc8182f3786c19817153265837692d527b0a1413554336074\": container with ID starting with b086b69771b0bf8dc8182f3786c19817153265837692d527b0a1413554336074 not found: ID does not exist" containerID="b086b69771b0bf8dc8182f3786c19817153265837692d527b0a1413554336074"
Sep 09 16:27:01.515230 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.515142    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b086b69771b0bf8dc8182f3786c19817153265837692d527b0a1413554336074"} err="failed to get container status \"b086b69771b0bf8dc8182f3786c19817153265837692d527b0a1413554336074\": rpc error: code = NotFound desc = could not find container \"b086b69771b0bf8dc8182f3786c19817153265837692d527b0a1413554336074\": container with ID starting with b086b69771b0bf8dc8182f3786c19817153265837692d527b0a1413554336074 not found: ID does not exist"
Sep 09 16:27:01.515230 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.515159    2576 scope.go:117] "RemoveContainer" containerID="ea8b560e5dfbb834d14e8b291c9779720197ee9d1f3e6bb4048bda3d6956d8ec"
Sep 09 16:27:01.524839 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.524638    2576 scope.go:117] "RemoveContainer" containerID="ea8b560e5dfbb834d14e8b291c9779720197ee9d1f3e6bb4048bda3d6956d8ec"
Sep 09 16:27:01.525541 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:27:01.525518    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8b560e5dfbb834d14e8b291c9779720197ee9d1f3e6bb4048bda3d6956d8ec\": container with ID starting with ea8b560e5dfbb834d14e8b291c9779720197ee9d1f3e6bb4048bda3d6956d8ec not found: ID does not exist" containerID="ea8b560e5dfbb834d14e8b291c9779720197ee9d1f3e6bb4048bda3d6956d8ec"
Sep 09 16:27:01.525620 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.525544    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8b560e5dfbb834d14e8b291c9779720197ee9d1f3e6bb4048bda3d6956d8ec"} err="failed to get container status \"ea8b560e5dfbb834d14e8b291c9779720197ee9d1f3e6bb4048bda3d6956d8ec\": rpc error: code = NotFound desc = could not find container \"ea8b560e5dfbb834d14e8b291c9779720197ee9d1f3e6bb4048bda3d6956d8ec\": container with ID starting with ea8b560e5dfbb834d14e8b291c9779720197ee9d1f3e6bb4048bda3d6956d8ec not found: ID does not exist"
Sep 09 16:27:01.531026 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.531005    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"]
Sep 09 16:27:01.535225 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.535206    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a375fa0-645bf88766-dcbf2"]
Sep 09 16:27:01.544372 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.544348    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"]
Sep 09 16:27:01.548158 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:01.548137    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-743aed-75b747c84b-4x2lz"]
Sep 09 16:27:02.553015 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:02.552983    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b603c64-fbd6-4401-8881-c5e65c46794b" path="/var/lib/kubelet/pods/2b603c64-fbd6-4401-8881-c5e65c46794b/volumes"
Sep 09 16:27:02.553389 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:02.553324    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90a40500-634d-44f6-a2d7-092442d873bb" path="/var/lib/kubelet/pods/90a40500-634d-44f6-a2d7-092442d873bb/volumes"
Sep 09 16:27:04.663253 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:04.663191    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td" podUID="7e0a2e41-53ac-463f-99f8-4c12f991d0a2" containerName="main" containerID="cri-o://c11d6df26429ebfcda2cdfffb2a3d37942dff95a182f7ceb055e7a04ae54ece9" gracePeriod=45
Sep 09 16:27:04.906415 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:04.906392    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:27:05.064283 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.064212    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-tmp-dir\") pod \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") "
Sep 09 16:27:05.064283 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.064244    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-dshm\") pod \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") "
Sep 09 16:27:05.064283 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.064283    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptdmj\" (UniqueName: \"kubernetes.io/projected/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-kube-api-access-ptdmj\") pod \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") "
Sep 09 16:27:05.064536 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.064314    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-model-cache\") pod \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") "
Sep 09 16:27:05.064536 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.064340    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-tls-certs\") pod \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") "
Sep 09 16:27:05.064536 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.064500    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-home\") pod \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\" (UID: \"7e0a2e41-53ac-463f-99f8-4c12f991d0a2\") "
Sep 09 16:27:05.064536 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.064508    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e0a2e41-53ac-463f-99f8-4c12f991d0a2" (UID: "7e0a2e41-53ac-463f-99f8-4c12f991d0a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:27:05.064753 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.064623    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e0a2e41-53ac-463f-99f8-4c12f991d0a2" (UID: "7e0a2e41-53ac-463f-99f8-4c12f991d0a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:27:05.064816 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.064763    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-home" (OuterVolumeSpecName: "home") pod "7e0a2e41-53ac-463f-99f8-4c12f991d0a2" (UID: "7e0a2e41-53ac-463f-99f8-4c12f991d0a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:27:05.064880 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.064872    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:05.064937 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.064887    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:05.064937 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.064898    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:05.066548 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.066529    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-dshm" (OuterVolumeSpecName: "dshm") pod "7e0a2e41-53ac-463f-99f8-4c12f991d0a2" (UID: "7e0a2e41-53ac-463f-99f8-4c12f991d0a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:27:05.066630 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.066611    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-kube-api-access-ptdmj" (OuterVolumeSpecName: "kube-api-access-ptdmj") pod "7e0a2e41-53ac-463f-99f8-4c12f991d0a2" (UID: "7e0a2e41-53ac-463f-99f8-4c12f991d0a2"). InnerVolumeSpecName "kube-api-access-ptdmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:27:05.066670 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.066632    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e0a2e41-53ac-463f-99f8-4c12f991d0a2" (UID: "7e0a2e41-53ac-463f-99f8-4c12f991d0a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:27:05.166165 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.166137    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptdmj\" (UniqueName: \"kubernetes.io/projected/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-kube-api-access-ptdmj\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:05.166165 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.166161    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:05.166165 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.166171    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0a2e41-53ac-463f-99f8-4c12f991d0a2-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:05.311633 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.311594    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d" podUID="1d09e129-d324-43d6-ab55-e3c974ac178e" containerName="main" containerID="cri-o://574430fd3b5aa166c679f10f0049ce82614ec4dffdb250d8691b8502b534faab" gracePeriod=45
Sep 09 16:27:05.529755 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.529718    2576 generic.go:356] "Generic (PLEG): container finished" podID="1d09e129-d324-43d6-ab55-e3c974ac178e" containerID="574430fd3b5aa166c679f10f0049ce82614ec4dffdb250d8691b8502b534faab" exitCode=0
Sep 09 16:27:05.529755 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.529740    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d" event={"ID":"1d09e129-d324-43d6-ab55-e3c974ac178e","Type":"ContainerDied","Data":"574430fd3b5aa166c679f10f0049ce82614ec4dffdb250d8691b8502b534faab"}
Sep 09 16:27:05.531201 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.531179    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e0a2e41-53ac-463f-99f8-4c12f991d0a2" containerID="c11d6df26429ebfcda2cdfffb2a3d37942dff95a182f7ceb055e7a04ae54ece9" exitCode=0
Sep 09 16:27:05.531326 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.531228    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"
Sep 09 16:27:05.531326 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.531264    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td" event={"ID":"7e0a2e41-53ac-463f-99f8-4c12f991d0a2","Type":"ContainerDied","Data":"c11d6df26429ebfcda2cdfffb2a3d37942dff95a182f7ceb055e7a04ae54ece9"}
Sep 09 16:27:05.531326 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.531305    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td" event={"ID":"7e0a2e41-53ac-463f-99f8-4c12f991d0a2","Type":"ContainerDied","Data":"ac4f42219e011de6455268136864540146b39c7b0330da3cee348e7bb7628a0b"}
Sep 09 16:27:05.531326 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.531322    2576 scope.go:117] "RemoveContainer" containerID="c11d6df26429ebfcda2cdfffb2a3d37942dff95a182f7ceb055e7a04ae54ece9"
Sep 09 16:27:05.540490 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.540475    2576 scope.go:117] "RemoveContainer" containerID="c11d6df26429ebfcda2cdfffb2a3d37942dff95a182f7ceb055e7a04ae54ece9"
Sep 09 16:27:05.540795 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:27:05.540775    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11d6df26429ebfcda2cdfffb2a3d37942dff95a182f7ceb055e7a04ae54ece9\": container with ID starting with c11d6df26429ebfcda2cdfffb2a3d37942dff95a182f7ceb055e7a04ae54ece9 not found: ID does not exist" containerID="c11d6df26429ebfcda2cdfffb2a3d37942dff95a182f7ceb055e7a04ae54ece9"
Sep 09 16:27:05.540890 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.540800    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11d6df26429ebfcda2cdfffb2a3d37942dff95a182f7ceb055e7a04ae54ece9"} err="failed to get container status \"c11d6df26429ebfcda2cdfffb2a3d37942dff95a182f7ceb055e7a04ae54ece9\": rpc error: code = NotFound desc = could not find container \"c11d6df26429ebfcda2cdfffb2a3d37942dff95a182f7ceb055e7a04ae54ece9\": container with ID starting with c11d6df26429ebfcda2cdfffb2a3d37942dff95a182f7ceb055e7a04ae54ece9 not found: ID does not exist"
Sep 09 16:27:05.549131 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.549114    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:27:05.551850 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.551832    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"]
Sep 09 16:27:05.555256 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.555237    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-b90990bf-kserve-54465bf78-xx6td"]
Sep 09 16:27:05.671347 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.671269    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-dshm\") pod \"1d09e129-d324-43d6-ab55-e3c974ac178e\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") "
Sep 09 16:27:05.671347 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.671316    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d09e129-d324-43d6-ab55-e3c974ac178e-tls-certs\") pod \"1d09e129-d324-43d6-ab55-e3c974ac178e\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") "
Sep 09 16:27:05.671347 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.671338    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-home\") pod \"1d09e129-d324-43d6-ab55-e3c974ac178e\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") "
Sep 09 16:27:05.671908 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.671359    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-model-cache\") pod \"1d09e129-d324-43d6-ab55-e3c974ac178e\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") "
Sep 09 16:27:05.671908 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.671485    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-tmp-dir\") pod \"1d09e129-d324-43d6-ab55-e3c974ac178e\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") "
Sep 09 16:27:05.671908 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.671547    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhx7r\" (UniqueName: \"kubernetes.io/projected/1d09e129-d324-43d6-ab55-e3c974ac178e-kube-api-access-zhx7r\") pod \"1d09e129-d324-43d6-ab55-e3c974ac178e\" (UID: \"1d09e129-d324-43d6-ab55-e3c974ac178e\") "
Sep 09 16:27:05.671908 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.671611    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d09e129-d324-43d6-ab55-e3c974ac178e" (UID: "1d09e129-d324-43d6-ab55-e3c974ac178e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:27:05.671908 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.671625    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-home" (OuterVolumeSpecName: "home") pod "1d09e129-d324-43d6-ab55-e3c974ac178e" (UID: "1d09e129-d324-43d6-ab55-e3c974ac178e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:27:05.671908 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.671677    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d09e129-d324-43d6-ab55-e3c974ac178e" (UID: "1d09e129-d324-43d6-ab55-e3c974ac178e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:27:05.672192 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.671952    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:05.672192 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.671974    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:05.672192 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.671990    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:05.673818 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.673785    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d09e129-d324-43d6-ab55-e3c974ac178e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d09e129-d324-43d6-ab55-e3c974ac178e" (UID: "1d09e129-d324-43d6-ab55-e3c974ac178e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:27:05.673947 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.673867    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d09e129-d324-43d6-ab55-e3c974ac178e-kube-api-access-zhx7r" (OuterVolumeSpecName: "kube-api-access-zhx7r") pod "1d09e129-d324-43d6-ab55-e3c974ac178e" (UID: "1d09e129-d324-43d6-ab55-e3c974ac178e"). InnerVolumeSpecName "kube-api-access-zhx7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:27:05.673947 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.673900    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-dshm" (OuterVolumeSpecName: "dshm") pod "1d09e129-d324-43d6-ab55-e3c974ac178e" (UID: "1d09e129-d324-43d6-ab55-e3c974ac178e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:27:05.773248 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.773207    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d09e129-d324-43d6-ab55-e3c974ac178e-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:05.773248 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.773240    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhx7r\" (UniqueName: \"kubernetes.io/projected/1d09e129-d324-43d6-ab55-e3c974ac178e-kube-api-access-zhx7r\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:05.773248 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:05.773252    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d09e129-d324-43d6-ab55-e3c974ac178e-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:06.542267 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:06.542237    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"
Sep 09 16:27:06.542445 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:06.542236    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d" event={"ID":"1d09e129-d324-43d6-ab55-e3c974ac178e","Type":"ContainerDied","Data":"340a26292bcfb85cf09e9e312732c0cb44a875b45901f88798fdcc06156ba5f7"}
Sep 09 16:27:06.542445 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:06.542374    2576 scope.go:117] "RemoveContainer" containerID="574430fd3b5aa166c679f10f0049ce82614ec4dffdb250d8691b8502b534faab"
Sep 09 16:27:06.552992 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:06.552967    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e0a2e41-53ac-463f-99f8-4c12f991d0a2" path="/var/lib/kubelet/pods/7e0a2e41-53ac-463f-99f8-4c12f991d0a2/volumes"
Sep 09 16:27:06.562633 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:06.562614    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"]
Sep 09 16:27:06.566678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:06.566655    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-743aed/autores-model-743aed-kserve-577bc6d87f-4hc4d"]
Sep 09 16:27:07.196147 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.196117    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"]
Sep 09 16:27:07.196512 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.196463    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e0a2e41-53ac-463f-99f8-4c12f991d0a2" containerName="main"
Sep 09 16:27:07.196512 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.196475    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e0a2e41-53ac-463f-99f8-4c12f991d0a2" containerName="main"
Sep 09 16:27:07.196512 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.196493    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90a40500-634d-44f6-a2d7-092442d873bb" containerName="payload-pre-processing"
Sep 09 16:27:07.196512 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.196501    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90a40500-634d-44f6-a2d7-092442d873bb" containerName="payload-pre-processing"
Sep 09 16:27:07.196512 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.196510    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b603c64-fbd6-4401-8881-c5e65c46794b" containerName="payload-pre-processing"
Sep 09 16:27:07.196726 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.196517    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b603c64-fbd6-4401-8881-c5e65c46794b" containerName="payload-pre-processing"
Sep 09 16:27:07.196726 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.196523    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d09e129-d324-43d6-ab55-e3c974ac178e" containerName="main"
Sep 09 16:27:07.196726 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.196530    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d09e129-d324-43d6-ab55-e3c974ac178e" containerName="main"
Sep 09 16:27:07.196726 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.196588    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d09e129-d324-43d6-ab55-e3c974ac178e" containerName="main"
Sep 09 16:27:07.196726 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.196596    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b603c64-fbd6-4401-8881-c5e65c46794b" containerName="payload-pre-processing"
Sep 09 16:27:07.196726 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.196602    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="90a40500-634d-44f6-a2d7-092442d873bb" containerName="payload-pre-processing"
Sep 09 16:27:07.196726 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.196608    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e0a2e41-53ac-463f-99f8-4c12f991d0a2" containerName="main"
Sep 09 16:27:07.200927 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.200903    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"
Sep 09 16:27:07.203201 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.203180    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2dc4f62b\""
Sep 09 16:27:07.203318 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.203226    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2dc4f62b-dockercfg-lgh8w\""
Sep 09 16:27:07.209212 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.209190    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"]
Sep 09 16:27:07.217189 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.217169    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"]
Sep 09 16:27:07.220258 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.220244    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"
Sep 09 16:27:07.228944 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.228922    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"]
Sep 09 16:27:07.388496 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.388458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9chlh\" (UniqueName: \"kubernetes.io/projected/064218b3-b74d-4b6e-a564-d91e4c261a37-kube-api-access-9chlh\") pod \"payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq\" (UID: \"064218b3-b74d-4b6e-a564-d91e4c261a37\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"
Sep 09 16:27:07.388666 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.388515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064218b3-b74d-4b6e-a564-d91e4c261a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq\" (UID: \"064218b3-b74d-4b6e-a564-d91e4c261a37\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"
Sep 09 16:27:07.388666 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.388587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2rsg\" (UniqueName: \"kubernetes.io/projected/4d7e0939-de6a-4a95-88c4-f62c96a09b82-kube-api-access-g2rsg\") pod \"payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw\" (UID: \"4d7e0939-de6a-4a95-88c4-f62c96a09b82\") " pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"
Sep 09 16:27:07.388666 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.388636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7e0939-de6a-4a95-88c4-f62c96a09b82-plugins-config-volume\") pod \"payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw\" (UID: \"4d7e0939-de6a-4a95-88c4-f62c96a09b82\") " pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"
Sep 09 16:27:07.489211 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.489179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064218b3-b74d-4b6e-a564-d91e4c261a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq\" (UID: \"064218b3-b74d-4b6e-a564-d91e4c261a37\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"
Sep 09 16:27:07.489401 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.489222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2rsg\" (UniqueName: \"kubernetes.io/projected/4d7e0939-de6a-4a95-88c4-f62c96a09b82-kube-api-access-g2rsg\") pod \"payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw\" (UID: \"4d7e0939-de6a-4a95-88c4-f62c96a09b82\") " pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"
Sep 09 16:27:07.489401 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.489244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7e0939-de6a-4a95-88c4-f62c96a09b82-plugins-config-volume\") pod \"payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw\" (UID: \"4d7e0939-de6a-4a95-88c4-f62c96a09b82\") " pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"
Sep 09 16:27:07.489401 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.489289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9chlh\" (UniqueName: \"kubernetes.io/projected/064218b3-b74d-4b6e-a564-d91e4c261a37-kube-api-access-9chlh\") pod \"payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq\" (UID: \"064218b3-b74d-4b6e-a564-d91e4c261a37\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"
Sep 09 16:27:07.489899 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.489875    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064218b3-b74d-4b6e-a564-d91e4c261a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq\" (UID: \"064218b3-b74d-4b6e-a564-d91e4c261a37\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"
Sep 09 16:27:07.490000 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.489973    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7e0939-de6a-4a95-88c4-f62c96a09b82-plugins-config-volume\") pod \"payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw\" (UID: \"4d7e0939-de6a-4a95-88c4-f62c96a09b82\") " pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"
Sep 09 16:27:07.497126 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.497103    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9chlh\" (UniqueName: \"kubernetes.io/projected/064218b3-b74d-4b6e-a564-d91e4c261a37-kube-api-access-9chlh\") pod \"payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq\" (UID: \"064218b3-b74d-4b6e-a564-d91e4c261a37\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"
Sep 09 16:27:07.497224 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.497134    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2rsg\" (UniqueName: \"kubernetes.io/projected/4d7e0939-de6a-4a95-88c4-f62c96a09b82-kube-api-access-g2rsg\") pod \"payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw\" (UID: \"4d7e0939-de6a-4a95-88c4-f62c96a09b82\") " pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"
Sep 09 16:27:07.511966 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.511943    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"
Sep 09 16:27:07.530240 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.530217    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"
Sep 09 16:27:07.785156 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.785077    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"]
Sep 09 16:27:07.787295 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:27:07.787268    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064218b3_b74d_4b6e_a564_d91e4c261a37.slice/crio-13381bdaafdfffbf5d428f87c4dfda0b285290231de2017ed61cad1f16012122 WatchSource:0}: Error finding container 13381bdaafdfffbf5d428f87c4dfda0b285290231de2017ed61cad1f16012122: Status 404 returned error can't find the container with id 13381bdaafdfffbf5d428f87c4dfda0b285290231de2017ed61cad1f16012122
Sep 09 16:27:07.811060 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:07.811037    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"]
Sep 09 16:27:07.813796 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:27:07.813772    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7e0939_de6a_4a95_88c4_f62c96a09b82.slice/crio-b8215f9a56dd46700c7116b405fb90d396a0ab5f5a4f3fd43f83afff681b1b7a WatchSource:0}: Error finding container b8215f9a56dd46700c7116b405fb90d396a0ab5f5a4f3fd43f83afff681b1b7a: Status 404 returned error can't find the container with id b8215f9a56dd46700c7116b405fb90d396a0ab5f5a4f3fd43f83afff681b1b7a
Sep 09 16:27:08.553852 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:08.553818    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d09e129-d324-43d6-ab55-e3c974ac178e" path="/var/lib/kubelet/pods/1d09e129-d324-43d6-ab55-e3c974ac178e/volumes"
Sep 09 16:27:08.557286 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:08.557257    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw" event={"ID":"4d7e0939-de6a-4a95-88c4-f62c96a09b82","Type":"ContainerStarted","Data":"0f1497f65a6358b1c2854a92b9496a0e3ad682161f5de909845322c0a3af29e6"}
Sep 09 16:27:08.557421 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:08.557309    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw" event={"ID":"4d7e0939-de6a-4a95-88c4-f62c96a09b82","Type":"ContainerStarted","Data":"b8215f9a56dd46700c7116b405fb90d396a0ab5f5a4f3fd43f83afff681b1b7a"}
Sep 09 16:27:08.557421 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:08.557337    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"
Sep 09 16:27:08.559121 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:08.559102    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq" event={"ID":"064218b3-b74d-4b6e-a564-d91e4c261a37","Type":"ContainerStarted","Data":"4d4f05f4d755892e93b563b3978b60fb6e4e15345a588bbae9e2db3ccd723992"}
Sep 09 16:27:08.559235 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:08.559141    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq" event={"ID":"064218b3-b74d-4b6e-a564-d91e4c261a37","Type":"ContainerStarted","Data":"13381bdaafdfffbf5d428f87c4dfda0b285290231de2017ed61cad1f16012122"}
Sep 09 16:27:08.559235 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:08.559173    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"
Sep 09 16:27:08.579874 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:08.579822    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw" podStartSLOduration=1.579807575 podStartE2EDuration="1.579807575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:27:08.579094553 +0000 UTC m=+1104.514712973" watchObservedRunningTime="2026-09-09 16:27:08.579807575 +0000 UTC m=+1104.515425992"
Sep 09 16:27:08.594844 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:08.594796    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq" podStartSLOduration=1.5947788539999999 podStartE2EDuration="1.594778854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:27:08.594510705 +0000 UTC m=+1104.530129135" watchObservedRunningTime="2026-09-09 16:27:08.594778854 +0000 UTC m=+1104.530397270"
Sep 09 16:27:14.518515 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.518476    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"]
Sep 09 16:27:14.521817 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.521796    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"
Sep 09 16:27:14.524244 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.524221    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-3ad4f1\""
Sep 09 16:27:14.530573 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.530554    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"]
Sep 09 16:27:14.545633 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.545609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhllf\" (UniqueName: \"kubernetes.io/projected/ddfeed14-97fb-49ea-8650-029a8d57b85b-kube-api-access-lhllf\") pod \"payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf\" (UID: \"ddfeed14-97fb-49ea-8650-029a8d57b85b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"
Sep 09 16:27:14.545780 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.545659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfeed14-97fb-49ea-8650-029a8d57b85b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf\" (UID: \"ddfeed14-97fb-49ea-8650-029a8d57b85b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"
Sep 09 16:27:14.561122 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.561096    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"]
Sep 09 16:27:14.564106 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.564093    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"
Sep 09 16:27:14.566532 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.566511    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-3ad4f1-dockercfg-hpg8s\""
Sep 09 16:27:14.574215 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.574186    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"]
Sep 09 16:27:14.646889 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.646850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfeed14-97fb-49ea-8650-029a8d57b85b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf\" (UID: \"ddfeed14-97fb-49ea-8650-029a8d57b85b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"
Sep 09 16:27:14.647050 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.646933    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq47t\" (UniqueName: \"kubernetes.io/projected/5be37c80-4067-44d3-9493-3ae867145d35-kube-api-access-dq47t\") pod \"payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j\" (UID: \"5be37c80-4067-44d3-9493-3ae867145d35\") " pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"
Sep 09 16:27:14.647050 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.646975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be37c80-4067-44d3-9493-3ae867145d35-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j\" (UID: \"5be37c80-4067-44d3-9493-3ae867145d35\") " pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"
Sep 09 16:27:14.647050 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.647003    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhllf\" (UniqueName: \"kubernetes.io/projected/ddfeed14-97fb-49ea-8650-029a8d57b85b-kube-api-access-lhllf\") pod \"payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf\" (UID: \"ddfeed14-97fb-49ea-8650-029a8d57b85b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"
Sep 09 16:27:14.647596 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.647572    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfeed14-97fb-49ea-8650-029a8d57b85b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf\" (UID: \"ddfeed14-97fb-49ea-8650-029a8d57b85b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"
Sep 09 16:27:14.654761 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.654737    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhllf\" (UniqueName: \"kubernetes.io/projected/ddfeed14-97fb-49ea-8650-029a8d57b85b-kube-api-access-lhllf\") pod \"payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf\" (UID: \"ddfeed14-97fb-49ea-8650-029a8d57b85b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"
Sep 09 16:27:14.747499 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.747462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq47t\" (UniqueName: \"kubernetes.io/projected/5be37c80-4067-44d3-9493-3ae867145d35-kube-api-access-dq47t\") pod \"payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j\" (UID: \"5be37c80-4067-44d3-9493-3ae867145d35\") " pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"
Sep 09 16:27:14.747641 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.747516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be37c80-4067-44d3-9493-3ae867145d35-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j\" (UID: \"5be37c80-4067-44d3-9493-3ae867145d35\") " pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"
Sep 09 16:27:14.748320 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.748300    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be37c80-4067-44d3-9493-3ae867145d35-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j\" (UID: \"5be37c80-4067-44d3-9493-3ae867145d35\") " pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"
Sep 09 16:27:14.755465 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.755442    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq47t\" (UniqueName: \"kubernetes.io/projected/5be37c80-4067-44d3-9493-3ae867145d35-kube-api-access-dq47t\") pod \"payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j\" (UID: \"5be37c80-4067-44d3-9493-3ae867145d35\") " pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"
Sep 09 16:27:14.835555 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.835474    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"
Sep 09 16:27:14.876723 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:14.876671    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"
Sep 09 16:27:15.111474 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:15.111448    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"]
Sep 09 16:27:15.113096 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:27:15.113068    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfeed14_97fb_49ea_8650_029a8d57b85b.slice/crio-e9fa47eff3eebe2889356612ab9bdc94d353bd9685452ccd24b4fdcfaa895369 WatchSource:0}: Error finding container e9fa47eff3eebe2889356612ab9bdc94d353bd9685452ccd24b4fdcfaa895369: Status 404 returned error can't find the container with id e9fa47eff3eebe2889356612ab9bdc94d353bd9685452ccd24b4fdcfaa895369
Sep 09 16:27:15.158436 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:15.158412    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"]
Sep 09 16:27:15.160567 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:27:15.160529    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be37c80_4067_44d3_9493_3ae867145d35.slice/crio-a047937548c3d9a1b7bae6ff980f5ce33bb560414840f0c81903bb20fb99ee55 WatchSource:0}: Error finding container a047937548c3d9a1b7bae6ff980f5ce33bb560414840f0c81903bb20fb99ee55: Status 404 returned error can't find the container with id a047937548c3d9a1b7bae6ff980f5ce33bb560414840f0c81903bb20fb99ee55
Sep 09 16:27:15.599953 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:15.599911    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j" event={"ID":"5be37c80-4067-44d3-9493-3ae867145d35","Type":"ContainerStarted","Data":"57a946fefdc262b8d917d09285a4fbf5d7d00231bd4b013ea74aefd3e19ccee8"}
Sep 09 16:27:15.600412 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:15.599973    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"
Sep 09 16:27:15.600412 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:15.600001    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j" event={"ID":"5be37c80-4067-44d3-9493-3ae867145d35","Type":"ContainerStarted","Data":"a047937548c3d9a1b7bae6ff980f5ce33bb560414840f0c81903bb20fb99ee55"}
Sep 09 16:27:15.601555 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:15.601530    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf" event={"ID":"ddfeed14-97fb-49ea-8650-029a8d57b85b","Type":"ContainerStarted","Data":"303dd355d945453863159b21369c3408ca69642fc55e2fb9a393127c2bb08662"}
Sep 09 16:27:15.601686 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:15.601569    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf" event={"ID":"ddfeed14-97fb-49ea-8650-029a8d57b85b","Type":"ContainerStarted","Data":"e9fa47eff3eebe2889356612ab9bdc94d353bd9685452ccd24b4fdcfaa895369"}
Sep 09 16:27:15.601686 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:15.601638    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"
Sep 09 16:27:15.617677 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:15.617620    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j" podStartSLOduration=1.61759875 podStartE2EDuration="1.61759875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:27:15.614271481 +0000 UTC m=+1111.549889915" watchObservedRunningTime="2026-09-09 16:27:15.61759875 +0000 UTC m=+1111.553217167"
Sep 09 16:27:15.629624 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:15.629569    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf" podStartSLOduration=1.6295539890000001 podStartE2EDuration="1.629553989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:27:15.627809329 +0000 UTC m=+1111.563427756" watchObservedRunningTime="2026-09-09 16:27:15.629553989 +0000 UTC m=+1111.565172404"
Sep 09 16:27:19.566554 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:19.566526    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"
Sep 09 16:27:19.566955 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:19.566682    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"
Sep 09 16:27:23.588059 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:23.588027    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"]
Sep 09 16:27:26.608168 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:26.608132    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"
Sep 09 16:27:26.608561 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:26.608219    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"
Sep 09 16:27:37.192095 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.192064    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"]
Sep 09 16:27:37.192631 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.192313    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw" podUID="4d7e0939-de6a-4a95-88c4-f62c96a09b82" containerName="payload-processing" containerID="cri-o://0f1497f65a6358b1c2854a92b9496a0e3ad682161f5de909845322c0a3af29e6" gracePeriod=30
Sep 09 16:27:37.200625 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.200596    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"]
Sep 09 16:27:37.201061 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.200997    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq" podUID="064218b3-b74d-4b6e-a564-d91e4c261a37" containerName="payload-pre-processing" containerID="cri-o://4d4f05f4d755892e93b563b3978b60fb6e4e15345a588bbae9e2db3ccd723992" gracePeriod=30
Sep 09 16:27:37.438471 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.438449    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"
Sep 09 16:27:37.466109 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.466085    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"
Sep 09 16:27:37.541711 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.541668    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064218b3-b74d-4b6e-a564-d91e4c261a37-plugins-config-volume\") pod \"064218b3-b74d-4b6e-a564-d91e4c261a37\" (UID: \"064218b3-b74d-4b6e-a564-d91e4c261a37\") "
Sep 09 16:27:37.541889 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.541745    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2rsg\" (UniqueName: \"kubernetes.io/projected/4d7e0939-de6a-4a95-88c4-f62c96a09b82-kube-api-access-g2rsg\") pod \"4d7e0939-de6a-4a95-88c4-f62c96a09b82\" (UID: \"4d7e0939-de6a-4a95-88c4-f62c96a09b82\") "
Sep 09 16:27:37.541889 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.541823    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7e0939-de6a-4a95-88c4-f62c96a09b82-plugins-config-volume\") pod \"4d7e0939-de6a-4a95-88c4-f62c96a09b82\" (UID: \"4d7e0939-de6a-4a95-88c4-f62c96a09b82\") "
Sep 09 16:27:37.541889 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.541861    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9chlh\" (UniqueName: \"kubernetes.io/projected/064218b3-b74d-4b6e-a564-d91e4c261a37-kube-api-access-9chlh\") pod \"064218b3-b74d-4b6e-a564-d91e4c261a37\" (UID: \"064218b3-b74d-4b6e-a564-d91e4c261a37\") "
Sep 09 16:27:37.542138 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.542113    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064218b3-b74d-4b6e-a564-d91e4c261a37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "064218b3-b74d-4b6e-a564-d91e4c261a37" (UID: "064218b3-b74d-4b6e-a564-d91e4c261a37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:27:37.542189 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.542169    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7e0939-de6a-4a95-88c4-f62c96a09b82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d7e0939-de6a-4a95-88c4-f62c96a09b82" (UID: "4d7e0939-de6a-4a95-88c4-f62c96a09b82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:27:37.543972 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.543948    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7e0939-de6a-4a95-88c4-f62c96a09b82-kube-api-access-g2rsg" (OuterVolumeSpecName: "kube-api-access-g2rsg") pod "4d7e0939-de6a-4a95-88c4-f62c96a09b82" (UID: "4d7e0939-de6a-4a95-88c4-f62c96a09b82"). InnerVolumeSpecName "kube-api-access-g2rsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:27:37.544079 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.544015    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064218b3-b74d-4b6e-a564-d91e4c261a37-kube-api-access-9chlh" (OuterVolumeSpecName: "kube-api-access-9chlh") pod "064218b3-b74d-4b6e-a564-d91e4c261a37" (UID: "064218b3-b74d-4b6e-a564-d91e4c261a37"). InnerVolumeSpecName "kube-api-access-9chlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:27:37.642933 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.642896    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7e0939-de6a-4a95-88c4-f62c96a09b82-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:37.642933 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.642931    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9chlh\" (UniqueName: \"kubernetes.io/projected/064218b3-b74d-4b6e-a564-d91e4c261a37-kube-api-access-9chlh\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:37.642933 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.642940    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064218b3-b74d-4b6e-a564-d91e4c261a37-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:37.642933 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.642949    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2rsg\" (UniqueName: \"kubernetes.io/projected/4d7e0939-de6a-4a95-88c4-f62c96a09b82-kube-api-access-g2rsg\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:37.704662 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.704598    2576 generic.go:356] "Generic (PLEG): container finished" podID="4d7e0939-de6a-4a95-88c4-f62c96a09b82" containerID="0f1497f65a6358b1c2854a92b9496a0e3ad682161f5de909845322c0a3af29e6" exitCode=0
Sep 09 16:27:37.704662 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.704649    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"
Sep 09 16:27:37.704848 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.704683    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw" event={"ID":"4d7e0939-de6a-4a95-88c4-f62c96a09b82","Type":"ContainerDied","Data":"0f1497f65a6358b1c2854a92b9496a0e3ad682161f5de909845322c0a3af29e6"}
Sep 09 16:27:37.704848 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.704770    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw" event={"ID":"4d7e0939-de6a-4a95-88c4-f62c96a09b82","Type":"ContainerDied","Data":"b8215f9a56dd46700c7116b405fb90d396a0ab5f5a4f3fd43f83afff681b1b7a"}
Sep 09 16:27:37.704848 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.704802    2576 scope.go:117] "RemoveContainer" containerID="0f1497f65a6358b1c2854a92b9496a0e3ad682161f5de909845322c0a3af29e6"
Sep 09 16:27:37.706157 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.706136    2576 generic.go:356] "Generic (PLEG): container finished" podID="064218b3-b74d-4b6e-a564-d91e4c261a37" containerID="4d4f05f4d755892e93b563b3978b60fb6e4e15345a588bbae9e2db3ccd723992" exitCode=0
Sep 09 16:27:37.706249 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.706177    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"
Sep 09 16:27:37.706249 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.706188    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq" event={"ID":"064218b3-b74d-4b6e-a564-d91e4c261a37","Type":"ContainerDied","Data":"4d4f05f4d755892e93b563b3978b60fb6e4e15345a588bbae9e2db3ccd723992"}
Sep 09 16:27:37.706249 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.706235    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq" event={"ID":"064218b3-b74d-4b6e-a564-d91e4c261a37","Type":"ContainerDied","Data":"13381bdaafdfffbf5d428f87c4dfda0b285290231de2017ed61cad1f16012122"}
Sep 09 16:27:37.715164 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.715147    2576 scope.go:117] "RemoveContainer" containerID="0f1497f65a6358b1c2854a92b9496a0e3ad682161f5de909845322c0a3af29e6"
Sep 09 16:27:37.715415 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:27:37.715399    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f1497f65a6358b1c2854a92b9496a0e3ad682161f5de909845322c0a3af29e6\": container with ID starting with 0f1497f65a6358b1c2854a92b9496a0e3ad682161f5de909845322c0a3af29e6 not found: ID does not exist" containerID="0f1497f65a6358b1c2854a92b9496a0e3ad682161f5de909845322c0a3af29e6"
Sep 09 16:27:37.715475 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.715424    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f1497f65a6358b1c2854a92b9496a0e3ad682161f5de909845322c0a3af29e6"} err="failed to get container status \"0f1497f65a6358b1c2854a92b9496a0e3ad682161f5de909845322c0a3af29e6\": rpc error: code = NotFound desc = could not find container \"0f1497f65a6358b1c2854a92b9496a0e3ad682161f5de909845322c0a3af29e6\": container with ID starting with 0f1497f65a6358b1c2854a92b9496a0e3ad682161f5de909845322c0a3af29e6 not found: ID does not exist"
Sep 09 16:27:37.715475 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.715440    2576 scope.go:117] "RemoveContainer" containerID="4d4f05f4d755892e93b563b3978b60fb6e4e15345a588bbae9e2db3ccd723992"
Sep 09 16:27:37.723083 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.723062    2576 scope.go:117] "RemoveContainer" containerID="4d4f05f4d755892e93b563b3978b60fb6e4e15345a588bbae9e2db3ccd723992"
Sep 09 16:27:37.723344 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:27:37.723324    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d4f05f4d755892e93b563b3978b60fb6e4e15345a588bbae9e2db3ccd723992\": container with ID starting with 4d4f05f4d755892e93b563b3978b60fb6e4e15345a588bbae9e2db3ccd723992 not found: ID does not exist" containerID="4d4f05f4d755892e93b563b3978b60fb6e4e15345a588bbae9e2db3ccd723992"
Sep 09 16:27:37.723405 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.723352    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4f05f4d755892e93b563b3978b60fb6e4e15345a588bbae9e2db3ccd723992"} err="failed to get container status \"4d4f05f4d755892e93b563b3978b60fb6e4e15345a588bbae9e2db3ccd723992\": rpc error: code = NotFound desc = could not find container \"4d4f05f4d755892e93b563b3978b60fb6e4e15345a588bbae9e2db3ccd723992\": container with ID starting with 4d4f05f4d755892e93b563b3978b60fb6e4e15345a588bbae9e2db3ccd723992 not found: ID does not exist"
Sep 09 16:27:37.726139 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.726118    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"]
Sep 09 16:27:37.731511 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.731490    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2dc4f62b-f665b47df-9nrvw"]
Sep 09 16:27:37.743077 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.743054    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"]
Sep 09 16:27:37.749048 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:37.749023    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2dc4f62b-9bd4ddfc4-f8npq"]
Sep 09 16:27:38.553028 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.552997    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="064218b3-b74d-4b6e-a564-d91e4c261a37" path="/var/lib/kubelet/pods/064218b3-b74d-4b6e-a564-d91e4c261a37/volumes"
Sep 09 16:27:38.553381 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.553340    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d7e0939-de6a-4a95-88c4-f62c96a09b82" path="/var/lib/kubelet/pods/4d7e0939-de6a-4a95-88c4-f62c96a09b82/volumes"
Sep 09 16:27:38.608862 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.608821    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn" podUID="85de2052-e98d-4697-8cea-d9208f88b113" containerName="main" containerID="cri-o://ca865a24590c3de1968a1c548fac8fc95bab0b17cf3b1745918adeef10ada377" gracePeriod=45
Sep 09 16:27:38.863424 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.863402    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:27:38.953216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.953187    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85de2052-e98d-4697-8cea-d9208f88b113-tls-certs\") pod \"85de2052-e98d-4697-8cea-d9208f88b113\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") "
Sep 09 16:27:38.953390 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.953229    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-tmp-dir\") pod \"85de2052-e98d-4697-8cea-d9208f88b113\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") "
Sep 09 16:27:38.953390 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.953255    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-dshm\") pod \"85de2052-e98d-4697-8cea-d9208f88b113\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") "
Sep 09 16:27:38.953390 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.953286    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-model-cache\") pod \"85de2052-e98d-4697-8cea-d9208f88b113\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") "
Sep 09 16:27:38.953390 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.953340    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-home\") pod \"85de2052-e98d-4697-8cea-d9208f88b113\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") "
Sep 09 16:27:38.953630 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.953411    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqp9v\" (UniqueName: \"kubernetes.io/projected/85de2052-e98d-4697-8cea-d9208f88b113-kube-api-access-zqp9v\") pod \"85de2052-e98d-4697-8cea-d9208f88b113\" (UID: \"85de2052-e98d-4697-8cea-d9208f88b113\") "
Sep 09 16:27:38.953630 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.953513    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85de2052-e98d-4697-8cea-d9208f88b113" (UID: "85de2052-e98d-4697-8cea-d9208f88b113"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:27:38.953630 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.953553    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-model-cache" (OuterVolumeSpecName: "model-cache") pod "85de2052-e98d-4697-8cea-d9208f88b113" (UID: "85de2052-e98d-4697-8cea-d9208f88b113"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:27:38.953769 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.953652    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-home" (OuterVolumeSpecName: "home") pod "85de2052-e98d-4697-8cea-d9208f88b113" (UID: "85de2052-e98d-4697-8cea-d9208f88b113"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:27:38.953815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.953791    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:38.953853 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.953812    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:38.953853 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.953826    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:38.955590 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.955566    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85de2052-e98d-4697-8cea-d9208f88b113-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85de2052-e98d-4697-8cea-d9208f88b113" (UID: "85de2052-e98d-4697-8cea-d9208f88b113"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:27:38.955710 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.955568    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85de2052-e98d-4697-8cea-d9208f88b113-kube-api-access-zqp9v" (OuterVolumeSpecName: "kube-api-access-zqp9v") pod "85de2052-e98d-4697-8cea-d9208f88b113" (UID: "85de2052-e98d-4697-8cea-d9208f88b113"). InnerVolumeSpecName "kube-api-access-zqp9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:27:38.955710 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:38.955630    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-dshm" (OuterVolumeSpecName: "dshm") pod "85de2052-e98d-4697-8cea-d9208f88b113" (UID: "85de2052-e98d-4697-8cea-d9208f88b113"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:27:39.054158 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:39.054131    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85de2052-e98d-4697-8cea-d9208f88b113-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:39.054158 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:39.054154    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85de2052-e98d-4697-8cea-d9208f88b113-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:39.054293 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:39.054162    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqp9v\" (UniqueName: \"kubernetes.io/projected/85de2052-e98d-4697-8cea-d9208f88b113-kube-api-access-zqp9v\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:27:39.718510 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:39.718471    2576 generic.go:356] "Generic (PLEG): container finished" podID="85de2052-e98d-4697-8cea-d9208f88b113" containerID="ca865a24590c3de1968a1c548fac8fc95bab0b17cf3b1745918adeef10ada377" exitCode=0
Sep 09 16:27:39.718969 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:39.718529    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"
Sep 09 16:27:39.718969 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:39.718563    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn" event={"ID":"85de2052-e98d-4697-8cea-d9208f88b113","Type":"ContainerDied","Data":"ca865a24590c3de1968a1c548fac8fc95bab0b17cf3b1745918adeef10ada377"}
Sep 09 16:27:39.718969 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:39.718612    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn" event={"ID":"85de2052-e98d-4697-8cea-d9208f88b113","Type":"ContainerDied","Data":"c50a0269bc5fb4bfc57ab3c3bd84d40a4e97abb3fbc6ba9dceace8631751f17a"}
Sep 09 16:27:39.718969 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:39.718637    2576 scope.go:117] "RemoveContainer" containerID="ca865a24590c3de1968a1c548fac8fc95bab0b17cf3b1745918adeef10ada377"
Sep 09 16:27:39.727644 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:39.727628    2576 scope.go:117] "RemoveContainer" containerID="ca865a24590c3de1968a1c548fac8fc95bab0b17cf3b1745918adeef10ada377"
Sep 09 16:27:39.727957 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:27:39.727936    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca865a24590c3de1968a1c548fac8fc95bab0b17cf3b1745918adeef10ada377\": container with ID starting with ca865a24590c3de1968a1c548fac8fc95bab0b17cf3b1745918adeef10ada377 not found: ID does not exist" containerID="ca865a24590c3de1968a1c548fac8fc95bab0b17cf3b1745918adeef10ada377"
Sep 09 16:27:39.728050 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:39.727962    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca865a24590c3de1968a1c548fac8fc95bab0b17cf3b1745918adeef10ada377"} err="failed to get container status \"ca865a24590c3de1968a1c548fac8fc95bab0b17cf3b1745918adeef10ada377\": rpc error: code = NotFound desc = could not find container \"ca865a24590c3de1968a1c548fac8fc95bab0b17cf3b1745918adeef10ada377\": container with ID starting with ca865a24590c3de1968a1c548fac8fc95bab0b17cf3b1745918adeef10ada377 not found: ID does not exist"
Sep 09 16:27:39.739157 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:39.739130    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"]
Sep 09 16:27:39.744627 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:39.744609    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-b90990bf-kserve-7fb4cb9cb4-pl8fn"]
Sep 09 16:27:40.554259 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:40.554226    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85de2052-e98d-4697-8cea-d9208f88b113" path="/var/lib/kubelet/pods/85de2052-e98d-4697-8cea-d9208f88b113/volumes"
Sep 09 16:27:43.304496 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.304468    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"]
Sep 09 16:27:43.304886 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.304797    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="064218b3-b74d-4b6e-a564-d91e4c261a37" containerName="payload-pre-processing"
Sep 09 16:27:43.304886 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.304808    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="064218b3-b74d-4b6e-a564-d91e4c261a37" containerName="payload-pre-processing"
Sep 09 16:27:43.304886 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.304817    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d7e0939-de6a-4a95-88c4-f62c96a09b82" containerName="payload-processing"
Sep 09 16:27:43.304886 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.304822    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d7e0939-de6a-4a95-88c4-f62c96a09b82" containerName="payload-processing"
Sep 09 16:27:43.304886 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.304835    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85de2052-e98d-4697-8cea-d9208f88b113" containerName="main"
Sep 09 16:27:43.304886 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.304841    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="85de2052-e98d-4697-8cea-d9208f88b113" containerName="main"
Sep 09 16:27:43.305081 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.304896    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="85de2052-e98d-4697-8cea-d9208f88b113" containerName="main"
Sep 09 16:27:43.305081 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.304903    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="064218b3-b74d-4b6e-a564-d91e4c261a37" containerName="payload-pre-processing"
Sep 09 16:27:43.305081 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.304908    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d7e0939-de6a-4a95-88c4-f62c96a09b82" containerName="payload-processing"
Sep 09 16:27:43.307955 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.307939    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.310245 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.310222    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c23332\"/\"openshift-service-ca.crt\""
Sep 09 16:27:43.311300 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.311279    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c23332\"/\"kube-root-ca.crt\""
Sep 09 16:27:43.311400 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.311359    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c23332\"/\"default-dockercfg-2n425\""
Sep 09 16:27:43.311453 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.311402    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c23332\"/\"test-model-c23332-kserve-self-signed-certs\""
Sep 09 16:27:43.315658 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.315620    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"]
Sep 09 16:27:43.389412 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.389375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gghf7\" (UniqueName: \"kubernetes.io/projected/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-kube-api-access-gghf7\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.389557 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.389514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-home\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.389594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.389554    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-tls-certs\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.389594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.389577    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-tmp-dir\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.389666 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.389594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-dshm\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.389666 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.389620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-model-cache\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.490531 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.490501    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-home\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.490531 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.490535    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-tls-certs\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.490790 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.490554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-tmp-dir\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.490790 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.490572    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-dshm\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.490790 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.490602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-model-cache\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.490790 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.490640    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gghf7\" (UniqueName: \"kubernetes.io/projected/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-kube-api-access-gghf7\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.491026 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.490998    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-home\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.491147 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.491045    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-tmp-dir\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.491147 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.491082    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-model-cache\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.493026 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.493007    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-dshm\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.493202 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.493184    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-tls-certs\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.498243 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.498222    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gghf7\" (UniqueName: \"kubernetes.io/projected/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-kube-api-access-gghf7\") pod \"test-model-c23332-kserve-6959f77cb9-br56g\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") " pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.618012 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.617951    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:43.890515 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:43.890484    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"]
Sep 09 16:27:43.893163 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:27:43.893138    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeee4d4f_e248_4375_9905_3d0a4ce81ef2.slice/crio-e72bb71dcf7b4f5c29c9e88a2e0ca3f4a72f48ddd32b707cc77178a5ea9e057e WatchSource:0}: Error finding container e72bb71dcf7b4f5c29c9e88a2e0ca3f4a72f48ddd32b707cc77178a5ea9e057e: Status 404 returned error can't find the container with id e72bb71dcf7b4f5c29c9e88a2e0ca3f4a72f48ddd32b707cc77178a5ea9e057e
Sep 09 16:27:44.741992 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:44.741956    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g" event={"ID":"eeee4d4f-e248-4375-9905-3d0a4ce81ef2","Type":"ContainerStarted","Data":"9b490bcb201e4f4457b5198dea25c8081a0b39bafe3c276777acb57494608304"}
Sep 09 16:27:44.741992 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:44.741995    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g" event={"ID":"eeee4d4f-e248-4375-9905-3d0a4ce81ef2","Type":"ContainerStarted","Data":"e72bb71dcf7b4f5c29c9e88a2e0ca3f4a72f48ddd32b707cc77178a5ea9e057e"}
Sep 09 16:27:44.758140 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:44.758096    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g" podStartSLOduration=1.758077706 podStartE2EDuration="1.758077706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:27:44.756572962 +0000 UTC m=+1140.692191375" watchObservedRunningTime="2026-09-09 16:27:44.758077706 +0000 UTC m=+1140.693696126"
Sep 09 16:27:53.618926 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:53.618895    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:53.619344 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:53.619033    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:53.631313 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:53.631290    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:53.793040 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:53.793010    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:27:54.595255 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.595225    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"]
Sep 09 16:27:54.599908 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.599885    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.603463 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.603432    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-3ad4f1\"/\"explicit-model-3ad4f1-kserve-self-signed-certs\""
Sep 09 16:27:54.603561 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.603435    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-3ad4f1\"/\"default-dockercfg-dxhqv\""
Sep 09 16:27:54.603561 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.603435    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-3ad4f1\"/\"kube-root-ca.crt\""
Sep 09 16:27:54.603561 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.603507    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-3ad4f1\"/\"openshift-service-ca.crt\""
Sep 09 16:27:54.606753 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.606728    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"]
Sep 09 16:27:54.697792 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.697759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwp2v\" (UniqueName: \"kubernetes.io/projected/ffac405f-bd29-4b83-8122-86705f741ba9-kube-api-access-hwp2v\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.698232 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.697808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffac405f-bd29-4b83-8122-86705f741ba9-tls-certs\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.698232 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.697925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-model-cache\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.698232 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.698002    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-home\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.698232 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.698041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-dshm\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.698232 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.698227    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-tmp-dir\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.799215 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.799185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-dshm\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.799393 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.799227    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-tmp-dir\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.799393 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.799268    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwp2v\" (UniqueName: \"kubernetes.io/projected/ffac405f-bd29-4b83-8122-86705f741ba9-kube-api-access-hwp2v\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.799393 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.799299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffac405f-bd29-4b83-8122-86705f741ba9-tls-certs\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.799393 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.799355    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-model-cache\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.799393 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.799387    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-home\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.799743 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.799718    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-tmp-dir\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.799829 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.799749    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-home\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.799829 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.799808    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-model-cache\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.801671 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.801653    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-dshm\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.802034 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.802019    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffac405f-bd29-4b83-8122-86705f741ba9-tls-certs\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.807856 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.807835    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwp2v\" (UniqueName: \"kubernetes.io/projected/ffac405f-bd29-4b83-8122-86705f741ba9-kube-api-access-hwp2v\") pod \"explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") " pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:54.910110 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:54.910022    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:27:55.179353 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:55.179239    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"]
Sep 09 16:27:55.182164 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:27:55.182134    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffac405f_bd29_4b83_8122_86705f741ba9.slice/crio-eca1bde8667fd193bca9efa24c5b48c8b044f552d73ca7edf4e36d04a2bd8b7f WatchSource:0}: Error finding container eca1bde8667fd193bca9efa24c5b48c8b044f552d73ca7edf4e36d04a2bd8b7f: Status 404 returned error can't find the container with id eca1bde8667fd193bca9efa24c5b48c8b044f552d73ca7edf4e36d04a2bd8b7f
Sep 09 16:27:55.791983 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:55.791943    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n" event={"ID":"ffac405f-bd29-4b83-8122-86705f741ba9","Type":"ContainerStarted","Data":"f562ceb210bdf9586a251be33dab67d62743251280c5a46ddff4a938ab61cd82"}
Sep 09 16:27:55.792406 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:55.791991    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n" event={"ID":"ffac405f-bd29-4b83-8122-86705f741ba9","Type":"ContainerStarted","Data":"eca1bde8667fd193bca9efa24c5b48c8b044f552d73ca7edf4e36d04a2bd8b7f"}
Sep 09 16:27:55.809756 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:27:55.809686    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n" podStartSLOduration=1.809670878 podStartE2EDuration="1.809670878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:27:55.808725229 +0000 UTC m=+1151.744343646" watchObservedRunningTime="2026-09-09 16:27:55.809670878 +0000 UTC m=+1151.745289294"
Sep 09 16:28:00.354650 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:00.354619    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"]
Sep 09 16:28:04.910153 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:04.910073    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:28:11.037093 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.037005    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"]
Sep 09 16:28:11.037605 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.037268    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j" podUID="5be37c80-4067-44d3-9493-3ae867145d35" containerName="payload-processing" containerID="cri-o://57a946fefdc262b8d917d09285a4fbf5d7d00231bd4b013ea74aefd3e19ccee8" gracePeriod=30
Sep 09 16:28:11.046509 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.046478    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"]
Sep 09 16:28:11.046737 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.046713    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf" podUID="ddfeed14-97fb-49ea-8650-029a8d57b85b" containerName="payload-pre-processing" containerID="cri-o://303dd355d945453863159b21369c3408ca69642fc55e2fb9a393127c2bb08662" gracePeriod=30
Sep 09 16:28:11.327678 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.327654    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"
Sep 09 16:28:11.330082 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.330066    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"
Sep 09 16:28:11.455690 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.455658    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq47t\" (UniqueName: \"kubernetes.io/projected/5be37c80-4067-44d3-9493-3ae867145d35-kube-api-access-dq47t\") pod \"5be37c80-4067-44d3-9493-3ae867145d35\" (UID: \"5be37c80-4067-44d3-9493-3ae867145d35\") "
Sep 09 16:28:11.455899 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.455734    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfeed14-97fb-49ea-8650-029a8d57b85b-plugins-config-volume\") pod \"ddfeed14-97fb-49ea-8650-029a8d57b85b\" (UID: \"ddfeed14-97fb-49ea-8650-029a8d57b85b\") "
Sep 09 16:28:11.455899 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.455769    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhllf\" (UniqueName: \"kubernetes.io/projected/ddfeed14-97fb-49ea-8650-029a8d57b85b-kube-api-access-lhllf\") pod \"ddfeed14-97fb-49ea-8650-029a8d57b85b\" (UID: \"ddfeed14-97fb-49ea-8650-029a8d57b85b\") "
Sep 09 16:28:11.455899 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.455796    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be37c80-4067-44d3-9493-3ae867145d35-plugins-config-volume\") pod \"5be37c80-4067-44d3-9493-3ae867145d35\" (UID: \"5be37c80-4067-44d3-9493-3ae867145d35\") "
Sep 09 16:28:11.456176 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.456149    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddfeed14-97fb-49ea-8650-029a8d57b85b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ddfeed14-97fb-49ea-8650-029a8d57b85b" (UID: "ddfeed14-97fb-49ea-8650-029a8d57b85b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:28:11.456240 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.456191    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be37c80-4067-44d3-9493-3ae867145d35-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5be37c80-4067-44d3-9493-3ae867145d35" (UID: "5be37c80-4067-44d3-9493-3ae867145d35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:28:11.457826 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.457800    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be37c80-4067-44d3-9493-3ae867145d35-kube-api-access-dq47t" (OuterVolumeSpecName: "kube-api-access-dq47t") pod "5be37c80-4067-44d3-9493-3ae867145d35" (UID: "5be37c80-4067-44d3-9493-3ae867145d35"). InnerVolumeSpecName "kube-api-access-dq47t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:28:11.457920 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.457886    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfeed14-97fb-49ea-8650-029a8d57b85b-kube-api-access-lhllf" (OuterVolumeSpecName: "kube-api-access-lhllf") pod "ddfeed14-97fb-49ea-8650-029a8d57b85b" (UID: "ddfeed14-97fb-49ea-8650-029a8d57b85b"). InnerVolumeSpecName "kube-api-access-lhllf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:28:11.557170 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.557103    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq47t\" (UniqueName: \"kubernetes.io/projected/5be37c80-4067-44d3-9493-3ae867145d35-kube-api-access-dq47t\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:11.557170 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.557131    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfeed14-97fb-49ea-8650-029a8d57b85b-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:11.557170 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.557140    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhllf\" (UniqueName: \"kubernetes.io/projected/ddfeed14-97fb-49ea-8650-029a8d57b85b-kube-api-access-lhllf\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:11.557170 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.557149    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be37c80-4067-44d3-9493-3ae867145d35-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:11.863018 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.862923    2576 generic.go:356] "Generic (PLEG): container finished" podID="ddfeed14-97fb-49ea-8650-029a8d57b85b" containerID="303dd355d945453863159b21369c3408ca69642fc55e2fb9a393127c2bb08662" exitCode=0
Sep 09 16:28:11.863018 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.862978    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"
Sep 09 16:28:11.863225 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.863016    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf" event={"ID":"ddfeed14-97fb-49ea-8650-029a8d57b85b","Type":"ContainerDied","Data":"303dd355d945453863159b21369c3408ca69642fc55e2fb9a393127c2bb08662"}
Sep 09 16:28:11.863225 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.863067    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf" event={"ID":"ddfeed14-97fb-49ea-8650-029a8d57b85b","Type":"ContainerDied","Data":"e9fa47eff3eebe2889356612ab9bdc94d353bd9685452ccd24b4fdcfaa895369"}
Sep 09 16:28:11.863225 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.863084    2576 scope.go:117] "RemoveContainer" containerID="303dd355d945453863159b21369c3408ca69642fc55e2fb9a393127c2bb08662"
Sep 09 16:28:11.864357 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.864337    2576 generic.go:356] "Generic (PLEG): container finished" podID="5be37c80-4067-44d3-9493-3ae867145d35" containerID="57a946fefdc262b8d917d09285a4fbf5d7d00231bd4b013ea74aefd3e19ccee8" exitCode=0
Sep 09 16:28:11.864444 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.864376    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j" event={"ID":"5be37c80-4067-44d3-9493-3ae867145d35","Type":"ContainerDied","Data":"57a946fefdc262b8d917d09285a4fbf5d7d00231bd4b013ea74aefd3e19ccee8"}
Sep 09 16:28:11.864444 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.864407    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j" event={"ID":"5be37c80-4067-44d3-9493-3ae867145d35","Type":"ContainerDied","Data":"a047937548c3d9a1b7bae6ff980f5ce33bb560414840f0c81903bb20fb99ee55"}
Sep 09 16:28:11.864444 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.864378    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"
Sep 09 16:28:11.872769 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.872601    2576 scope.go:117] "RemoveContainer" containerID="303dd355d945453863159b21369c3408ca69642fc55e2fb9a393127c2bb08662"
Sep 09 16:28:11.872928 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:11.872910    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303dd355d945453863159b21369c3408ca69642fc55e2fb9a393127c2bb08662\": container with ID starting with 303dd355d945453863159b21369c3408ca69642fc55e2fb9a393127c2bb08662 not found: ID does not exist" containerID="303dd355d945453863159b21369c3408ca69642fc55e2fb9a393127c2bb08662"
Sep 09 16:28:11.872994 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.872935    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303dd355d945453863159b21369c3408ca69642fc55e2fb9a393127c2bb08662"} err="failed to get container status \"303dd355d945453863159b21369c3408ca69642fc55e2fb9a393127c2bb08662\": rpc error: code = NotFound desc = could not find container \"303dd355d945453863159b21369c3408ca69642fc55e2fb9a393127c2bb08662\": container with ID starting with 303dd355d945453863159b21369c3408ca69642fc55e2fb9a393127c2bb08662 not found: ID does not exist"
Sep 09 16:28:11.872994 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.872951    2576 scope.go:117] "RemoveContainer" containerID="57a946fefdc262b8d917d09285a4fbf5d7d00231bd4b013ea74aefd3e19ccee8"
Sep 09 16:28:11.880807 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.880790    2576 scope.go:117] "RemoveContainer" containerID="57a946fefdc262b8d917d09285a4fbf5d7d00231bd4b013ea74aefd3e19ccee8"
Sep 09 16:28:11.881050 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:11.881034    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a946fefdc262b8d917d09285a4fbf5d7d00231bd4b013ea74aefd3e19ccee8\": container with ID starting with 57a946fefdc262b8d917d09285a4fbf5d7d00231bd4b013ea74aefd3e19ccee8 not found: ID does not exist" containerID="57a946fefdc262b8d917d09285a4fbf5d7d00231bd4b013ea74aefd3e19ccee8"
Sep 09 16:28:11.881113 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.881055    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a946fefdc262b8d917d09285a4fbf5d7d00231bd4b013ea74aefd3e19ccee8"} err="failed to get container status \"57a946fefdc262b8d917d09285a4fbf5d7d00231bd4b013ea74aefd3e19ccee8\": rpc error: code = NotFound desc = could not find container \"57a946fefdc262b8d917d09285a4fbf5d7d00231bd4b013ea74aefd3e19ccee8\": container with ID starting with 57a946fefdc262b8d917d09285a4fbf5d7d00231bd4b013ea74aefd3e19ccee8 not found: ID does not exist"
Sep 09 16:28:11.885737 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.885714    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"]
Sep 09 16:28:11.891872 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.891854    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3ad4f1-7c6f78cb4b-xzpxf"]
Sep 09 16:28:11.902420 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.902400    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"]
Sep 09 16:28:11.905960 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:11.905942    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3ad4f1-58bb8f6f69-mz49j"]
Sep 09 16:28:12.553144 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:12.553111    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5be37c80-4067-44d3-9493-3ae867145d35" path="/var/lib/kubelet/pods/5be37c80-4067-44d3-9493-3ae867145d35/volumes"
Sep 09 16:28:12.553500 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:12.553440    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddfeed14-97fb-49ea-8650-029a8d57b85b" path="/var/lib/kubelet/pods/ddfeed14-97fb-49ea-8650-029a8d57b85b/volumes"
Sep 09 16:28:15.374938 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.374898    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n" podUID="ffac405f-bd29-4b83-8122-86705f741ba9" containerName="main" containerID="cri-o://f562ceb210bdf9586a251be33dab67d62743251280c5a46ddff4a938ab61cd82" gracePeriod=45
Sep 09 16:28:15.626241 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.626181    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:28:15.696650 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.696613    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-dshm\") pod \"ffac405f-bd29-4b83-8122-86705f741ba9\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") "
Sep 09 16:28:15.696859 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.696718    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-tmp-dir\") pod \"ffac405f-bd29-4b83-8122-86705f741ba9\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") "
Sep 09 16:28:15.696859 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.696745    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-home\") pod \"ffac405f-bd29-4b83-8122-86705f741ba9\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") "
Sep 09 16:28:15.696859 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.696788    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffac405f-bd29-4b83-8122-86705f741ba9-tls-certs\") pod \"ffac405f-bd29-4b83-8122-86705f741ba9\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") "
Sep 09 16:28:15.696859 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.696828    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwp2v\" (UniqueName: \"kubernetes.io/projected/ffac405f-bd29-4b83-8122-86705f741ba9-kube-api-access-hwp2v\") pod \"ffac405f-bd29-4b83-8122-86705f741ba9\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") "
Sep 09 16:28:15.696859 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.696848    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-model-cache\") pod \"ffac405f-bd29-4b83-8122-86705f741ba9\" (UID: \"ffac405f-bd29-4b83-8122-86705f741ba9\") "
Sep 09 16:28:15.697141 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.696993    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffac405f-bd29-4b83-8122-86705f741ba9" (UID: "ffac405f-bd29-4b83-8122-86705f741ba9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:28:15.697141 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.697007    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-home" (OuterVolumeSpecName: "home") pod "ffac405f-bd29-4b83-8122-86705f741ba9" (UID: "ffac405f-bd29-4b83-8122-86705f741ba9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:28:15.697141 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.697114    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffac405f-bd29-4b83-8122-86705f741ba9" (UID: "ffac405f-bd29-4b83-8122-86705f741ba9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:28:15.697328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.697313    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:15.697380 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.697333    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:15.697380 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.697346    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:15.699227 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.699203    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffac405f-bd29-4b83-8122-86705f741ba9-kube-api-access-hwp2v" (OuterVolumeSpecName: "kube-api-access-hwp2v") pod "ffac405f-bd29-4b83-8122-86705f741ba9" (UID: "ffac405f-bd29-4b83-8122-86705f741ba9"). InnerVolumeSpecName "kube-api-access-hwp2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:28:15.699498 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.699474    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-dshm" (OuterVolumeSpecName: "dshm") pod "ffac405f-bd29-4b83-8122-86705f741ba9" (UID: "ffac405f-bd29-4b83-8122-86705f741ba9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:28:15.699498 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.699483    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffac405f-bd29-4b83-8122-86705f741ba9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffac405f-bd29-4b83-8122-86705f741ba9" (UID: "ffac405f-bd29-4b83-8122-86705f741ba9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:28:15.798298 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.798269    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffac405f-bd29-4b83-8122-86705f741ba9-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:15.798298 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.798292    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffac405f-bd29-4b83-8122-86705f741ba9-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:15.798298 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.798303    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwp2v\" (UniqueName: \"kubernetes.io/projected/ffac405f-bd29-4b83-8122-86705f741ba9-kube-api-access-hwp2v\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:15.884515 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.884432    2576 generic.go:356] "Generic (PLEG): container finished" podID="ffac405f-bd29-4b83-8122-86705f741ba9" containerID="f562ceb210bdf9586a251be33dab67d62743251280c5a46ddff4a938ab61cd82" exitCode=0
Sep 09 16:28:15.884515 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.884482    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"
Sep 09 16:28:15.884515 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.884488    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n" event={"ID":"ffac405f-bd29-4b83-8122-86705f741ba9","Type":"ContainerDied","Data":"f562ceb210bdf9586a251be33dab67d62743251280c5a46ddff4a938ab61cd82"}
Sep 09 16:28:15.884793 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.884520    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n" event={"ID":"ffac405f-bd29-4b83-8122-86705f741ba9","Type":"ContainerDied","Data":"eca1bde8667fd193bca9efa24c5b48c8b044f552d73ca7edf4e36d04a2bd8b7f"}
Sep 09 16:28:15.884793 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.884539    2576 scope.go:117] "RemoveContainer" containerID="f562ceb210bdf9586a251be33dab67d62743251280c5a46ddff4a938ab61cd82"
Sep 09 16:28:15.893430 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.893400    2576 scope.go:117] "RemoveContainer" containerID="f562ceb210bdf9586a251be33dab67d62743251280c5a46ddff4a938ab61cd82"
Sep 09 16:28:15.893787 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:15.893674    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f562ceb210bdf9586a251be33dab67d62743251280c5a46ddff4a938ab61cd82\": container with ID starting with f562ceb210bdf9586a251be33dab67d62743251280c5a46ddff4a938ab61cd82 not found: ID does not exist" containerID="f562ceb210bdf9586a251be33dab67d62743251280c5a46ddff4a938ab61cd82"
Sep 09 16:28:15.893787 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.893721    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f562ceb210bdf9586a251be33dab67d62743251280c5a46ddff4a938ab61cd82"} err="failed to get container status \"f562ceb210bdf9586a251be33dab67d62743251280c5a46ddff4a938ab61cd82\": rpc error: code = NotFound desc = could not find container \"f562ceb210bdf9586a251be33dab67d62743251280c5a46ddff4a938ab61cd82\": container with ID starting with f562ceb210bdf9586a251be33dab67d62743251280c5a46ddff4a938ab61cd82 not found: ID does not exist"
Sep 09 16:28:15.905183 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.905164    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"]
Sep 09 16:28:15.909447 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:15.909418    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-3ad4f1/explicit-model-3ad4f1-kserve-7d7c9b55ff-6nz4n"]
Sep 09 16:28:16.552715 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:16.552674    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffac405f-bd29-4b83-8122-86705f741ba9" path="/var/lib/kubelet/pods/ffac405f-bd29-4b83-8122-86705f741ba9/volumes"
Sep 09 16:28:30.709121 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.709088    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"]
Sep 09 16:28:30.709673 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.709435    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddfeed14-97fb-49ea-8650-029a8d57b85b" containerName="payload-pre-processing"
Sep 09 16:28:30.709673 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.709447    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddfeed14-97fb-49ea-8650-029a8d57b85b" containerName="payload-pre-processing"
Sep 09 16:28:30.709673 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.709463    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffac405f-bd29-4b83-8122-86705f741ba9" containerName="main"
Sep 09 16:28:30.709673 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.709468    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffac405f-bd29-4b83-8122-86705f741ba9" containerName="main"
Sep 09 16:28:30.709673 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.709480    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be37c80-4067-44d3-9493-3ae867145d35" containerName="payload-processing"
Sep 09 16:28:30.709673 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.709486    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be37c80-4067-44d3-9493-3ae867145d35" containerName="payload-processing"
Sep 09 16:28:30.709673 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.709537    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be37c80-4067-44d3-9493-3ae867145d35" containerName="payload-processing"
Sep 09 16:28:30.709673 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.709544    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddfeed14-97fb-49ea-8650-029a8d57b85b" containerName="payload-pre-processing"
Sep 09 16:28:30.709673 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.709551    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffac405f-bd29-4b83-8122-86705f741ba9" containerName="main"
Sep 09 16:28:30.713676 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.713655    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.717095 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.717074    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-639363\"/\"openshift-service-ca.crt\""
Sep 09 16:28:30.717198 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.717137    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-639363\"/\"default-dockercfg-cjxgt\""
Sep 09 16:28:30.717198 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.717074    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-639363\"/\"noten-model-639363-kserve-self-signed-certs\""
Sep 09 16:28:30.717198 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.717083    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-639363\"/\"kube-root-ca.crt\""
Sep 09 16:28:30.722416 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.722395    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"]
Sep 09 16:28:30.828011 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.827982    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-model-cache\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.828011 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.828011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzgl8\" (UniqueName: \"kubernetes.io/projected/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-kube-api-access-dzgl8\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.828182 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.828034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.828182 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.828064    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-home\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.828182 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.828130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-dshm\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.828281 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.828189    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tmp-dir\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.929180 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.929156    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-model-cache\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.929341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.929185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzgl8\" (UniqueName: \"kubernetes.io/projected/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-kube-api-access-dzgl8\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.929341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.929211    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.929341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.929245    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-home\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.929341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.929283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-dshm\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.929539 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.929347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tmp-dir\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.929608 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.929588    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-model-cache\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.929666 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.929620    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-home\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.929751 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.929688    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tmp-dir\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.931576 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.931557    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-dshm\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.931840 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.931823    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:30.937593 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:30.937568    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzgl8\" (UniqueName: \"kubernetes.io/projected/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-kube-api-access-dzgl8\") pod \"noten-model-639363-kserve-6796bcf787-w6skm\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") " pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:31.023807 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:31.023786    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:31.294279 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:31.294064    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"]
Sep 09 16:28:31.296767 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:28:31.296737    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14ea0b3a_9400_49c5_b246_ae5d1fbeb6bf.slice/crio-057fda88c5b9bff0e620f7e955161850e02b7918d3b163bc12e080ffe6db35b9 WatchSource:0}: Error finding container 057fda88c5b9bff0e620f7e955161850e02b7918d3b163bc12e080ffe6db35b9: Status 404 returned error can't find the container with id 057fda88c5b9bff0e620f7e955161850e02b7918d3b163bc12e080ffe6db35b9
Sep 09 16:28:31.956420 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:31.956383    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm" event={"ID":"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf","Type":"ContainerStarted","Data":"358a898dd50e0a2a931640c3c5fc4ace815110fa10abc7568b6fc5639348a4ed"}
Sep 09 16:28:31.956838 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:31.956428    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm" event={"ID":"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf","Type":"ContainerStarted","Data":"057fda88c5b9bff0e620f7e955161850e02b7918d3b163bc12e080ffe6db35b9"}
Sep 09 16:28:31.971998 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:31.971952    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm" podStartSLOduration=1.97193789 podStartE2EDuration="1.97193789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:28:31.971893637 +0000 UTC m=+1187.907512057" watchObservedRunningTime="2026-09-09 16:28:31.97193789 +0000 UTC m=+1187.907556305"
Sep 09 16:28:32.724728 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:32.724672    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"]
Sep 09 16:28:33.049721 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:33.049610    2576 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-639363/noten-model-639363-kserve-self-signed-certs: secret "noten-model-639363-kserve-self-signed-certs" not found
Sep 09 16:28:33.049721 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:33.049685    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs podName:14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf nodeName:}" failed. No retries permitted until 2026-09-09 16:28:33.549666562 +0000 UTC m=+1189.485284956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs") pod "noten-model-639363-kserve-6796bcf787-w6skm" (UID: "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf") : secret "noten-model-639363-kserve-self-signed-certs" not found
Sep 09 16:28:33.553161 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:33.553132    2576 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-639363/noten-model-639363-kserve-self-signed-certs: secret "noten-model-639363-kserve-self-signed-certs" not found
Sep 09 16:28:33.553322 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:33.553194    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs podName:14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf nodeName:}" failed. No retries permitted until 2026-09-09 16:28:34.553180285 +0000 UTC m=+1190.488798680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs") pod "noten-model-639363-kserve-6796bcf787-w6skm" (UID: "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf") : secret "noten-model-639363-kserve-self-signed-certs" not found
Sep 09 16:28:34.561497 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:34.561465    2576 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-639363/noten-model-639363-kserve-self-signed-certs: secret "noten-model-639363-kserve-self-signed-certs" not found
Sep 09 16:28:34.561906 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:34.561523    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs podName:14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf nodeName:}" failed. No retries permitted until 2026-09-09 16:28:36.561510339 +0000 UTC m=+1192.497128733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs") pod "noten-model-639363-kserve-6796bcf787-w6skm" (UID: "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf") : secret "noten-model-639363-kserve-self-signed-certs" not found
Sep 09 16:28:36.580796 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:36.580763    2576 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-639363/noten-model-639363-kserve-self-signed-certs: secret "noten-model-639363-kserve-self-signed-certs" not found
Sep 09 16:28:36.581200 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:36.580836    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs podName:14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf nodeName:}" failed. No retries permitted until 2026-09-09 16:28:40.580822276 +0000 UTC m=+1196.516440670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs") pod "noten-model-639363-kserve-6796bcf787-w6skm" (UID: "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf") : secret "noten-model-639363-kserve-self-signed-certs" not found
Sep 09 16:28:40.614525 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:40.614496    2576 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-639363/noten-model-639363-kserve-self-signed-certs: secret "noten-model-639363-kserve-self-signed-certs" not found
Sep 09 16:28:40.614951 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:40.614569    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs podName:14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf nodeName:}" failed. No retries permitted until 2026-09-09 16:28:48.614553505 +0000 UTC m=+1204.550171902 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs") pod "noten-model-639363-kserve-6796bcf787-w6skm" (UID: "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf") : secret "noten-model-639363-kserve-self-signed-certs" not found
Sep 09 16:28:41.024884 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:41.024848    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:48.684833 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:48.684794    2576 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-639363/noten-model-639363-kserve-self-signed-certs: secret "noten-model-639363-kserve-self-signed-certs" not found
Sep 09 16:28:48.685275 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:48.684869    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs podName:14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf nodeName:}" failed. No retries permitted until 2026-09-09 16:29:04.684853343 +0000 UTC m=+1220.620471736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs") pod "noten-model-639363-kserve-6796bcf787-w6skm" (UID: "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf") : secret "noten-model-639363-kserve-self-signed-certs" not found
Sep 09 16:28:48.984594 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:48.984533    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm" podUID="14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf" containerName="main" containerID="cri-o://358a898dd50e0a2a931640c3c5fc4ace815110fa10abc7568b6fc5639348a4ed" gracePeriod=45
Sep 09 16:28:49.226173 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.226143    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:49.289486 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.289394    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzgl8\" (UniqueName: \"kubernetes.io/projected/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-kube-api-access-dzgl8\") pod \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") "
Sep 09 16:28:49.289486 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.289438    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-dshm\") pod \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") "
Sep 09 16:28:49.289741 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.289516    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-model-cache\") pod \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") "
Sep 09 16:28:49.289741 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.289560    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tmp-dir\") pod \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") "
Sep 09 16:28:49.289741 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.289598    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-home\") pod \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") "
Sep 09 16:28:49.289741 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.289638    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs\") pod \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\" (UID: \"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf\") "
Sep 09 16:28:49.289936 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.289845    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf" (UID: "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:28:49.289936 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.289880    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf" (UID: "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:28:49.290014 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.289931    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-home" (OuterVolumeSpecName: "home") pod "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf" (UID: "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:28:49.290014 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.289974    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:49.291902 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.291871    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-dshm" (OuterVolumeSpecName: "dshm") pod "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf" (UID: "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:28:49.292004 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.291913    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-kube-api-access-dzgl8" (OuterVolumeSpecName: "kube-api-access-dzgl8") pod "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf" (UID: "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf"). InnerVolumeSpecName "kube-api-access-dzgl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:28:49.292004 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.291971    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf" (UID: "14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:28:49.390400 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.390368    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:49.390400 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.390396    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzgl8\" (UniqueName: \"kubernetes.io/projected/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-kube-api-access-dzgl8\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:49.390400 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.390406    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:49.390630 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.390415    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:49.390630 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:49.390424    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:28:50.037426 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:50.037391    2576 generic.go:356] "Generic (PLEG): container finished" podID="14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf" containerID="358a898dd50e0a2a931640c3c5fc4ace815110fa10abc7568b6fc5639348a4ed" exitCode=0
Sep 09 16:28:50.037866 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:50.037447    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"
Sep 09 16:28:50.037866 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:50.037451    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm" event={"ID":"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf","Type":"ContainerDied","Data":"358a898dd50e0a2a931640c3c5fc4ace815110fa10abc7568b6fc5639348a4ed"}
Sep 09 16:28:50.037866 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:50.037555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm" event={"ID":"14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf","Type":"ContainerDied","Data":"057fda88c5b9bff0e620f7e955161850e02b7918d3b163bc12e080ffe6db35b9"}
Sep 09 16:28:50.037866 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:50.037573    2576 scope.go:117] "RemoveContainer" containerID="358a898dd50e0a2a931640c3c5fc4ace815110fa10abc7568b6fc5639348a4ed"
Sep 09 16:28:50.046959 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:50.046847    2576 scope.go:117] "RemoveContainer" containerID="358a898dd50e0a2a931640c3c5fc4ace815110fa10abc7568b6fc5639348a4ed"
Sep 09 16:28:50.047105 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:28:50.047087    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358a898dd50e0a2a931640c3c5fc4ace815110fa10abc7568b6fc5639348a4ed\": container with ID starting with 358a898dd50e0a2a931640c3c5fc4ace815110fa10abc7568b6fc5639348a4ed not found: ID does not exist" containerID="358a898dd50e0a2a931640c3c5fc4ace815110fa10abc7568b6fc5639348a4ed"
Sep 09 16:28:50.047155 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:50.047111    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358a898dd50e0a2a931640c3c5fc4ace815110fa10abc7568b6fc5639348a4ed"} err="failed to get container status \"358a898dd50e0a2a931640c3c5fc4ace815110fa10abc7568b6fc5639348a4ed\": rpc error: code = NotFound desc = could not find container \"358a898dd50e0a2a931640c3c5fc4ace815110fa10abc7568b6fc5639348a4ed\": container with ID starting with 358a898dd50e0a2a931640c3c5fc4ace815110fa10abc7568b6fc5639348a4ed not found: ID does not exist"
Sep 09 16:28:50.058004 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:50.057983    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"]
Sep 09 16:28:50.060311 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:50.060291    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-639363/noten-model-639363-kserve-6796bcf787-w6skm"]
Sep 09 16:28:50.553057 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:50.553024    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf" path="/var/lib/kubelet/pods/14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf/volumes"
Sep 09 16:28:52.689315 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:28:52.689282    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"]
Sep 09 16:29:03.408809 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.408775    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"]
Sep 09 16:29:03.409247 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.408978    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj" podUID="03b12307-d8e3-4da6-8c4a-494b3a3b0862" containerName="payload-processing" containerID="cri-o://f826118c2d42efd976509107c12f815ea61a052eb5ebefc5501fa6d5bb5b46a0" gracePeriod=30
Sep 09 16:29:03.416792 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.416759    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"]
Sep 09 16:29:03.416982 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.416962    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l" podUID="f55f996a-d2aa-4fc8-a38d-798a7053e0b5" containerName="payload-pre-processing" containerID="cri-o://9ee47ee28fd64e37c1811133418034870a940664eda4e683b13cf42311ce78e0" gracePeriod=30
Sep 09 16:29:03.688753 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.688732    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"
Sep 09 16:29:03.691424 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.691408    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"
Sep 09 16:29:03.812278 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.812249    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55f996a-d2aa-4fc8-a38d-798a7053e0b5-plugins-config-volume\") pod \"f55f996a-d2aa-4fc8-a38d-798a7053e0b5\" (UID: \"f55f996a-d2aa-4fc8-a38d-798a7053e0b5\") "
Sep 09 16:29:03.812469 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.812291    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b12307-d8e3-4da6-8c4a-494b3a3b0862-plugins-config-volume\") pod \"03b12307-d8e3-4da6-8c4a-494b3a3b0862\" (UID: \"03b12307-d8e3-4da6-8c4a-494b3a3b0862\") "
Sep 09 16:29:03.812469 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.812346    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dlsn\" (UniqueName: \"kubernetes.io/projected/f55f996a-d2aa-4fc8-a38d-798a7053e0b5-kube-api-access-2dlsn\") pod \"f55f996a-d2aa-4fc8-a38d-798a7053e0b5\" (UID: \"f55f996a-d2aa-4fc8-a38d-798a7053e0b5\") "
Sep 09 16:29:03.812469 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.812447    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf4rn\" (UniqueName: \"kubernetes.io/projected/03b12307-d8e3-4da6-8c4a-494b3a3b0862-kube-api-access-xf4rn\") pod \"03b12307-d8e3-4da6-8c4a-494b3a3b0862\" (UID: \"03b12307-d8e3-4da6-8c4a-494b3a3b0862\") "
Sep 09 16:29:03.812691 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.812668    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b12307-d8e3-4da6-8c4a-494b3a3b0862-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03b12307-d8e3-4da6-8c4a-494b3a3b0862" (UID: "03b12307-d8e3-4da6-8c4a-494b3a3b0862"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:29:03.812691 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.812683    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55f996a-d2aa-4fc8-a38d-798a7053e0b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f55f996a-d2aa-4fc8-a38d-798a7053e0b5" (UID: "f55f996a-d2aa-4fc8-a38d-798a7053e0b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:29:03.812828 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.812810    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55f996a-d2aa-4fc8-a38d-798a7053e0b5-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:03.812871 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.812834    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b12307-d8e3-4da6-8c4a-494b3a3b0862-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:03.814666 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.814645    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55f996a-d2aa-4fc8-a38d-798a7053e0b5-kube-api-access-2dlsn" (OuterVolumeSpecName: "kube-api-access-2dlsn") pod "f55f996a-d2aa-4fc8-a38d-798a7053e0b5" (UID: "f55f996a-d2aa-4fc8-a38d-798a7053e0b5"). InnerVolumeSpecName "kube-api-access-2dlsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:29:03.814742 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.814676    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b12307-d8e3-4da6-8c4a-494b3a3b0862-kube-api-access-xf4rn" (OuterVolumeSpecName: "kube-api-access-xf4rn") pod "03b12307-d8e3-4da6-8c4a-494b3a3b0862" (UID: "03b12307-d8e3-4da6-8c4a-494b3a3b0862"). InnerVolumeSpecName "kube-api-access-xf4rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:29:03.913437 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.913358    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dlsn\" (UniqueName: \"kubernetes.io/projected/f55f996a-d2aa-4fc8-a38d-798a7053e0b5-kube-api-access-2dlsn\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:03.913437 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:03.913386    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf4rn\" (UniqueName: \"kubernetes.io/projected/03b12307-d8e3-4da6-8c4a-494b3a3b0862-kube-api-access-xf4rn\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:04.102858 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.102826    2576 generic.go:356] "Generic (PLEG): container finished" podID="03b12307-d8e3-4da6-8c4a-494b3a3b0862" containerID="f826118c2d42efd976509107c12f815ea61a052eb5ebefc5501fa6d5bb5b46a0" exitCode=0
Sep 09 16:29:04.103048 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.102865    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"
Sep 09 16:29:04.103048 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.102905    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj" event={"ID":"03b12307-d8e3-4da6-8c4a-494b3a3b0862","Type":"ContainerDied","Data":"f826118c2d42efd976509107c12f815ea61a052eb5ebefc5501fa6d5bb5b46a0"}
Sep 09 16:29:04.103048 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.102961    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj" event={"ID":"03b12307-d8e3-4da6-8c4a-494b3a3b0862","Type":"ContainerDied","Data":"5dced465bb465175e0a87dede7a5352dcd4c242383e9626d1c60ead53d03511d"}
Sep 09 16:29:04.103048 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.102991    2576 scope.go:117] "RemoveContainer" containerID="f826118c2d42efd976509107c12f815ea61a052eb5ebefc5501fa6d5bb5b46a0"
Sep 09 16:29:04.104317 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.104290    2576 generic.go:356] "Generic (PLEG): container finished" podID="f55f996a-d2aa-4fc8-a38d-798a7053e0b5" containerID="9ee47ee28fd64e37c1811133418034870a940664eda4e683b13cf42311ce78e0" exitCode=0
Sep 09 16:29:04.104452 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.104321    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l" event={"ID":"f55f996a-d2aa-4fc8-a38d-798a7053e0b5","Type":"ContainerDied","Data":"9ee47ee28fd64e37c1811133418034870a940664eda4e683b13cf42311ce78e0"}
Sep 09 16:29:04.104452 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.104376    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l" event={"ID":"f55f996a-d2aa-4fc8-a38d-798a7053e0b5","Type":"ContainerDied","Data":"445b3d099e9d20d6173e2638ad2df7b834eda5b9063d5be9674856405255c61d"}
Sep 09 16:29:04.104452 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.104335    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"
Sep 09 16:29:04.112843 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.112785    2576 scope.go:117] "RemoveContainer" containerID="f826118c2d42efd976509107c12f815ea61a052eb5ebefc5501fa6d5bb5b46a0"
Sep 09 16:29:04.113158 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:29:04.113135    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f826118c2d42efd976509107c12f815ea61a052eb5ebefc5501fa6d5bb5b46a0\": container with ID starting with f826118c2d42efd976509107c12f815ea61a052eb5ebefc5501fa6d5bb5b46a0 not found: ID does not exist" containerID="f826118c2d42efd976509107c12f815ea61a052eb5ebefc5501fa6d5bb5b46a0"
Sep 09 16:29:04.113232 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.113171    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f826118c2d42efd976509107c12f815ea61a052eb5ebefc5501fa6d5bb5b46a0"} err="failed to get container status \"f826118c2d42efd976509107c12f815ea61a052eb5ebefc5501fa6d5bb5b46a0\": rpc error: code = NotFound desc = could not find container \"f826118c2d42efd976509107c12f815ea61a052eb5ebefc5501fa6d5bb5b46a0\": container with ID starting with f826118c2d42efd976509107c12f815ea61a052eb5ebefc5501fa6d5bb5b46a0 not found: ID does not exist"
Sep 09 16:29:04.113232 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.113193    2576 scope.go:117] "RemoveContainer" containerID="9ee47ee28fd64e37c1811133418034870a940664eda4e683b13cf42311ce78e0"
Sep 09 16:29:04.121097 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.121078    2576 scope.go:117] "RemoveContainer" containerID="9ee47ee28fd64e37c1811133418034870a940664eda4e683b13cf42311ce78e0"
Sep 09 16:29:04.121360 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:29:04.121339    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee47ee28fd64e37c1811133418034870a940664eda4e683b13cf42311ce78e0\": container with ID starting with 9ee47ee28fd64e37c1811133418034870a940664eda4e683b13cf42311ce78e0 not found: ID does not exist" containerID="9ee47ee28fd64e37c1811133418034870a940664eda4e683b13cf42311ce78e0"
Sep 09 16:29:04.121420 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.121369    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee47ee28fd64e37c1811133418034870a940664eda4e683b13cf42311ce78e0"} err="failed to get container status \"9ee47ee28fd64e37c1811133418034870a940664eda4e683b13cf42311ce78e0\": rpc error: code = NotFound desc = could not find container \"9ee47ee28fd64e37c1811133418034870a940664eda4e683b13cf42311ce78e0\": container with ID starting with 9ee47ee28fd64e37c1811133418034870a940664eda4e683b13cf42311ce78e0 not found: ID does not exist"
Sep 09 16:29:04.128726 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.128251    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"]
Sep 09 16:29:04.132933 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.132907    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6d44b6-56bdb45c8b-sq29l"]
Sep 09 16:29:04.143610 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.143582    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"]
Sep 09 16:29:04.147394 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.147369    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6d44b6-7474d9fb69-b4tvj"]
Sep 09 16:29:04.553137 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.553103    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03b12307-d8e3-4da6-8c4a-494b3a3b0862" path="/var/lib/kubelet/pods/03b12307-d8e3-4da6-8c4a-494b3a3b0862/volumes"
Sep 09 16:29:04.553545 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:04.553451    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f55f996a-d2aa-4fc8-a38d-798a7053e0b5" path="/var/lib/kubelet/pods/f55f996a-d2aa-4fc8-a38d-798a7053e0b5/volumes"
Sep 09 16:29:07.713660 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:07.713600    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g" podUID="eeee4d4f-e248-4375-9905-3d0a4ce81ef2" containerName="main" containerID="cri-o://9b490bcb201e4f4457b5198dea25c8081a0b39bafe3c276777acb57494608304" gracePeriod=45
Sep 09 16:29:07.957795 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:07.957771    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:29:08.048240 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.048207    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-model-cache\") pod \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") "
Sep 09 16:29:08.048240 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.048246    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-tmp-dir\") pod \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") "
Sep 09 16:29:08.048496 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.048277    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gghf7\" (UniqueName: \"kubernetes.io/projected/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-kube-api-access-gghf7\") pod \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") "
Sep 09 16:29:08.048496 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.048303    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-dshm\") pod \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") "
Sep 09 16:29:08.048496 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.048426    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-home\") pod \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") "
Sep 09 16:29:08.048496 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.048467    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-tls-certs\") pod \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\" (UID: \"eeee4d4f-e248-4375-9905-3d0a4ce81ef2\") "
Sep 09 16:29:08.048729 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.048573    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eeee4d4f-e248-4375-9905-3d0a4ce81ef2" (UID: "eeee4d4f-e248-4375-9905-3d0a4ce81ef2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:29:08.048729 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.048585    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-model-cache" (OuterVolumeSpecName: "model-cache") pod "eeee4d4f-e248-4375-9905-3d0a4ce81ef2" (UID: "eeee4d4f-e248-4375-9905-3d0a4ce81ef2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:29:08.048809 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.048739    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-home" (OuterVolumeSpecName: "home") pod "eeee4d4f-e248-4375-9905-3d0a4ce81ef2" (UID: "eeee4d4f-e248-4375-9905-3d0a4ce81ef2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:29:08.048809 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.048763    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:08.048809 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.048778    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:08.050707 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.050666    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eeee4d4f-e248-4375-9905-3d0a4ce81ef2" (UID: "eeee4d4f-e248-4375-9905-3d0a4ce81ef2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:29:08.050707 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.050675    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-dshm" (OuterVolumeSpecName: "dshm") pod "eeee4d4f-e248-4375-9905-3d0a4ce81ef2" (UID: "eeee4d4f-e248-4375-9905-3d0a4ce81ef2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:29:08.050870 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.050684    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-kube-api-access-gghf7" (OuterVolumeSpecName: "kube-api-access-gghf7") pod "eeee4d4f-e248-4375-9905-3d0a4ce81ef2" (UID: "eeee4d4f-e248-4375-9905-3d0a4ce81ef2"). InnerVolumeSpecName "kube-api-access-gghf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:29:08.125439 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.125403    2576 generic.go:356] "Generic (PLEG): container finished" podID="eeee4d4f-e248-4375-9905-3d0a4ce81ef2" containerID="9b490bcb201e4f4457b5198dea25c8081a0b39bafe3c276777acb57494608304" exitCode=0
Sep 09 16:29:08.125611 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.125450    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"
Sep 09 16:29:08.125611 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.125493    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g" event={"ID":"eeee4d4f-e248-4375-9905-3d0a4ce81ef2","Type":"ContainerDied","Data":"9b490bcb201e4f4457b5198dea25c8081a0b39bafe3c276777acb57494608304"}
Sep 09 16:29:08.125611 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.125536    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g" event={"ID":"eeee4d4f-e248-4375-9905-3d0a4ce81ef2","Type":"ContainerDied","Data":"e72bb71dcf7b4f5c29c9e88a2e0ca3f4a72f48ddd32b707cc77178a5ea9e057e"}
Sep 09 16:29:08.125611 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.125552    2576 scope.go:117] "RemoveContainer" containerID="9b490bcb201e4f4457b5198dea25c8081a0b39bafe3c276777acb57494608304"
Sep 09 16:29:08.134612 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.134583    2576 scope.go:117] "RemoveContainer" containerID="9b490bcb201e4f4457b5198dea25c8081a0b39bafe3c276777acb57494608304"
Sep 09 16:29:08.134887 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:29:08.134870    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b490bcb201e4f4457b5198dea25c8081a0b39bafe3c276777acb57494608304\": container with ID starting with 9b490bcb201e4f4457b5198dea25c8081a0b39bafe3c276777acb57494608304 not found: ID does not exist" containerID="9b490bcb201e4f4457b5198dea25c8081a0b39bafe3c276777acb57494608304"
Sep 09 16:29:08.134934 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.134896    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b490bcb201e4f4457b5198dea25c8081a0b39bafe3c276777acb57494608304"} err="failed to get container status \"9b490bcb201e4f4457b5198dea25c8081a0b39bafe3c276777acb57494608304\": rpc error: code = NotFound desc = could not find container \"9b490bcb201e4f4457b5198dea25c8081a0b39bafe3c276777acb57494608304\": container with ID starting with 9b490bcb201e4f4457b5198dea25c8081a0b39bafe3c276777acb57494608304 not found: ID does not exist"
Sep 09 16:29:08.145079 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.145040    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"]
Sep 09 16:29:08.149299 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.149191    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gghf7\" (UniqueName: \"kubernetes.io/projected/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-kube-api-access-gghf7\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:08.149299 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.149216    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:08.149299 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.149230    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:08.149299 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.149240    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeee4d4f-e248-4375-9905-3d0a4ce81ef2-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:08.150926 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.150894    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c23332/test-model-c23332-kserve-6959f77cb9-br56g"]
Sep 09 16:29:08.553459 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:08.553425    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeee4d4f-e248-4375-9905-3d0a4ce81ef2" path="/var/lib/kubelet/pods/eeee4d4f-e248-4375-9905-3d0a4ce81ef2/volumes"
Sep 09 16:29:18.374443 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.374406    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"]
Sep 09 16:29:18.374911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.374690    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b12307-d8e3-4da6-8c4a-494b3a3b0862" containerName="payload-processing"
Sep 09 16:29:18.374911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.374717    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b12307-d8e3-4da6-8c4a-494b3a3b0862" containerName="payload-processing"
Sep 09 16:29:18.374911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.374737    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf" containerName="main"
Sep 09 16:29:18.374911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.374742    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf" containerName="main"
Sep 09 16:29:18.374911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.374749    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55f996a-d2aa-4fc8-a38d-798a7053e0b5" containerName="payload-pre-processing"
Sep 09 16:29:18.374911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.374755    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55f996a-d2aa-4fc8-a38d-798a7053e0b5" containerName="payload-pre-processing"
Sep 09 16:29:18.374911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.374760    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeee4d4f-e248-4375-9905-3d0a4ce81ef2" containerName="main"
Sep 09 16:29:18.374911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.374766    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeee4d4f-e248-4375-9905-3d0a4ce81ef2" containerName="main"
Sep 09 16:29:18.374911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.374816    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ea0b3a-9400-49c5-b246-ae5d1fbeb6bf" containerName="main"
Sep 09 16:29:18.374911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.374822    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeee4d4f-e248-4375-9905-3d0a4ce81ef2" containerName="main"
Sep 09 16:29:18.374911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.374828    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55f996a-d2aa-4fc8-a38d-798a7053e0b5" containerName="payload-pre-processing"
Sep 09 16:29:18.374911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.374833    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b12307-d8e3-4da6-8c4a-494b3a3b0862" containerName="payload-processing"
Sep 09 16:29:18.377645 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.377629    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"
Sep 09 16:29:18.380123 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.380098    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-13b02364-dockercfg-bdzt7\""
Sep 09 16:29:18.380256 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.380127    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-13b02364\""
Sep 09 16:29:18.388627 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.388598    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"]
Sep 09 16:29:18.409531 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.409497    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"]
Sep 09 16:29:18.412766 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.412750    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"
Sep 09 16:29:18.421938 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.421912    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"]
Sep 09 16:29:18.539505 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.539472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbbj4\" (UniqueName: \"kubernetes.io/projected/5457463b-6eb5-4150-89a8-6981100c6530-kube-api-access-qbbj4\") pod \"payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg\" (UID: \"5457463b-6eb5-4150-89a8-6981100c6530\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"
Sep 09 16:29:18.539689 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.539527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5457463b-6eb5-4150-89a8-6981100c6530-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg\" (UID: \"5457463b-6eb5-4150-89a8-6981100c6530\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"
Sep 09 16:29:18.539689 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.539555    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67524555-833c-4f09-8fe6-b81fc49d648b-plugins-config-volume\") pod \"payload-processing-e2e-mt-13b02364-d4d456597-7dq7q\" (UID: \"67524555-833c-4f09-8fe6-b81fc49d648b\") " pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"
Sep 09 16:29:18.539689 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.539577    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s25nj\" (UniqueName: \"kubernetes.io/projected/67524555-833c-4f09-8fe6-b81fc49d648b-kube-api-access-s25nj\") pod \"payload-processing-e2e-mt-13b02364-d4d456597-7dq7q\" (UID: \"67524555-833c-4f09-8fe6-b81fc49d648b\") " pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"
Sep 09 16:29:18.640473 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.640374    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbbj4\" (UniqueName: \"kubernetes.io/projected/5457463b-6eb5-4150-89a8-6981100c6530-kube-api-access-qbbj4\") pod \"payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg\" (UID: \"5457463b-6eb5-4150-89a8-6981100c6530\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"
Sep 09 16:29:18.640473 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.640416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5457463b-6eb5-4150-89a8-6981100c6530-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg\" (UID: \"5457463b-6eb5-4150-89a8-6981100c6530\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"
Sep 09 16:29:18.640753 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.640579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67524555-833c-4f09-8fe6-b81fc49d648b-plugins-config-volume\") pod \"payload-processing-e2e-mt-13b02364-d4d456597-7dq7q\" (UID: \"67524555-833c-4f09-8fe6-b81fc49d648b\") " pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"
Sep 09 16:29:18.640753 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.640628    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s25nj\" (UniqueName: \"kubernetes.io/projected/67524555-833c-4f09-8fe6-b81fc49d648b-kube-api-access-s25nj\") pod \"payload-processing-e2e-mt-13b02364-d4d456597-7dq7q\" (UID: \"67524555-833c-4f09-8fe6-b81fc49d648b\") " pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"
Sep 09 16:29:18.641176 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.641157    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5457463b-6eb5-4150-89a8-6981100c6530-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg\" (UID: \"5457463b-6eb5-4150-89a8-6981100c6530\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"
Sep 09 16:29:18.641352 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.641333    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67524555-833c-4f09-8fe6-b81fc49d648b-plugins-config-volume\") pod \"payload-processing-e2e-mt-13b02364-d4d456597-7dq7q\" (UID: \"67524555-833c-4f09-8fe6-b81fc49d648b\") " pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"
Sep 09 16:29:18.648465 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.648432    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s25nj\" (UniqueName: \"kubernetes.io/projected/67524555-833c-4f09-8fe6-b81fc49d648b-kube-api-access-s25nj\") pod \"payload-processing-e2e-mt-13b02364-d4d456597-7dq7q\" (UID: \"67524555-833c-4f09-8fe6-b81fc49d648b\") " pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"
Sep 09 16:29:18.648576 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.648530    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbbj4\" (UniqueName: \"kubernetes.io/projected/5457463b-6eb5-4150-89a8-6981100c6530-kube-api-access-qbbj4\") pod \"payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg\" (UID: \"5457463b-6eb5-4150-89a8-6981100c6530\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"
Sep 09 16:29:18.687441 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.687401    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"
Sep 09 16:29:18.724233 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.724173    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"
Sep 09 16:29:18.959595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:18.959509    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"]
Sep 09 16:29:18.962357 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:29:18.962329    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5457463b_6eb5_4150_89a8_6981100c6530.slice/crio-b160ebfb248056441087ed43e8336b24454e0f9b15960fb7fea0b8c685a3f90a WatchSource:0}: Error finding container b160ebfb248056441087ed43e8336b24454e0f9b15960fb7fea0b8c685a3f90a: Status 404 returned error can't find the container with id b160ebfb248056441087ed43e8336b24454e0f9b15960fb7fea0b8c685a3f90a
Sep 09 16:29:19.006779 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:19.006751    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"]
Sep 09 16:29:19.013099 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:29:19.013076    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67524555_833c_4f09_8fe6_b81fc49d648b.slice/crio-b61e22f94d47e061d29f22ce8fddc1965c978e8f5ec1c80dacc5a3e36f876270 WatchSource:0}: Error finding container b61e22f94d47e061d29f22ce8fddc1965c978e8f5ec1c80dacc5a3e36f876270: Status 404 returned error can't find the container with id b61e22f94d47e061d29f22ce8fddc1965c978e8f5ec1c80dacc5a3e36f876270
Sep 09 16:29:19.172795 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:19.172732    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg" event={"ID":"5457463b-6eb5-4150-89a8-6981100c6530","Type":"ContainerStarted","Data":"44a5cf00a623ab5f0cddfd8a9fc3bcdf617b65d94fa21620a051d997a43f96e3"}
Sep 09 16:29:19.172988 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:19.172893    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"
Sep 09 16:29:19.172988 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:19.172936    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg" event={"ID":"5457463b-6eb5-4150-89a8-6981100c6530","Type":"ContainerStarted","Data":"b160ebfb248056441087ed43e8336b24454e0f9b15960fb7fea0b8c685a3f90a"}
Sep 09 16:29:19.174619 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:19.174595    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q" event={"ID":"67524555-833c-4f09-8fe6-b81fc49d648b","Type":"ContainerStarted","Data":"0ce32297691839e86153429ecec3f8871f765231beaed0f47bd3fcf173e1ec93"}
Sep 09 16:29:19.174750 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:19.174641    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q" event={"ID":"67524555-833c-4f09-8fe6-b81fc49d648b","Type":"ContainerStarted","Data":"b61e22f94d47e061d29f22ce8fddc1965c978e8f5ec1c80dacc5a3e36f876270"}
Sep 09 16:29:19.174750 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:19.174680    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"
Sep 09 16:29:19.190347 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:19.190289    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg" podStartSLOduration=1.190271234 podStartE2EDuration="1.190271234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:29:19.188300481 +0000 UTC m=+1235.123918958" watchObservedRunningTime="2026-09-09 16:29:19.190271234 +0000 UTC m=+1235.125889652"
Sep 09 16:29:19.204230 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:19.204185    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q" podStartSLOduration=1.204170241 podStartE2EDuration="1.204170241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:29:19.202674676 +0000 UTC m=+1235.138293102" watchObservedRunningTime="2026-09-09 16:29:19.204170241 +0000 UTC m=+1235.139788677"
Sep 09 16:29:21.103721 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.103674    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"]
Sep 09 16:29:21.104178 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.103925    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j" podUID="8adb5336-488c-4988-93c8-6c160ab5889d" containerName="payload-processing" containerID="cri-o://58820fd27f571e87a5a2796e6d4a79ee5b5118de19118cb0b2ad9f3ab861edeb" gracePeriod=30
Sep 09 16:29:21.109546 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.109520    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"]
Sep 09 16:29:21.109772 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.109726    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q" podUID="987395c3-26da-4d13-8564-f329160dde1d" containerName="payload-pre-processing" containerID="cri-o://4b274b3800974c703dffd3a4ac38bb2c0751e3e079ea9fc0860e5a36d59a35bb" gracePeriod=30
Sep 09 16:29:21.389438 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.389409    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"
Sep 09 16:29:21.392599 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.392577    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"
Sep 09 16:29:21.567457 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.567424    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987395c3-26da-4d13-8564-f329160dde1d-plugins-config-volume\") pod \"987395c3-26da-4d13-8564-f329160dde1d\" (UID: \"987395c3-26da-4d13-8564-f329160dde1d\") "
Sep 09 16:29:21.567663 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.567485    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8adb5336-488c-4988-93c8-6c160ab5889d-plugins-config-volume\") pod \"8adb5336-488c-4988-93c8-6c160ab5889d\" (UID: \"8adb5336-488c-4988-93c8-6c160ab5889d\") "
Sep 09 16:29:21.567663 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.567549    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7xjk\" (UniqueName: \"kubernetes.io/projected/8adb5336-488c-4988-93c8-6c160ab5889d-kube-api-access-t7xjk\") pod \"8adb5336-488c-4988-93c8-6c160ab5889d\" (UID: \"8adb5336-488c-4988-93c8-6c160ab5889d\") "
Sep 09 16:29:21.567817 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.567680    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cfdb\" (UniqueName: \"kubernetes.io/projected/987395c3-26da-4d13-8564-f329160dde1d-kube-api-access-9cfdb\") pod \"987395c3-26da-4d13-8564-f329160dde1d\" (UID: \"987395c3-26da-4d13-8564-f329160dde1d\") "
Sep 09 16:29:21.567877 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.567847    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987395c3-26da-4d13-8564-f329160dde1d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "987395c3-26da-4d13-8564-f329160dde1d" (UID: "987395c3-26da-4d13-8564-f329160dde1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:29:21.567950 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.567919    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8adb5336-488c-4988-93c8-6c160ab5889d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8adb5336-488c-4988-93c8-6c160ab5889d" (UID: "8adb5336-488c-4988-93c8-6c160ab5889d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:29:21.568102 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.568072    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987395c3-26da-4d13-8564-f329160dde1d-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:21.568102 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.568098    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8adb5336-488c-4988-93c8-6c160ab5889d-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:21.570086 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.570062    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8adb5336-488c-4988-93c8-6c160ab5889d-kube-api-access-t7xjk" (OuterVolumeSpecName: "kube-api-access-t7xjk") pod "8adb5336-488c-4988-93c8-6c160ab5889d" (UID: "8adb5336-488c-4988-93c8-6c160ab5889d"). InnerVolumeSpecName "kube-api-access-t7xjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:29:21.570190 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.570175    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987395c3-26da-4d13-8564-f329160dde1d-kube-api-access-9cfdb" (OuterVolumeSpecName: "kube-api-access-9cfdb") pod "987395c3-26da-4d13-8564-f329160dde1d" (UID: "987395c3-26da-4d13-8564-f329160dde1d"). InnerVolumeSpecName "kube-api-access-9cfdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:29:21.669252 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.669150    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7xjk\" (UniqueName: \"kubernetes.io/projected/8adb5336-488c-4988-93c8-6c160ab5889d-kube-api-access-t7xjk\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:21.669252 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:21.669188    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cfdb\" (UniqueName: \"kubernetes.io/projected/987395c3-26da-4d13-8564-f329160dde1d-kube-api-access-9cfdb\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:22.192997 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.192957    2576 generic.go:356] "Generic (PLEG): container finished" podID="987395c3-26da-4d13-8564-f329160dde1d" containerID="4b274b3800974c703dffd3a4ac38bb2c0751e3e079ea9fc0860e5a36d59a35bb" exitCode=0
Sep 09 16:29:22.193473 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.193018    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q" event={"ID":"987395c3-26da-4d13-8564-f329160dde1d","Type":"ContainerDied","Data":"4b274b3800974c703dffd3a4ac38bb2c0751e3e079ea9fc0860e5a36d59a35bb"}
Sep 09 16:29:22.193473 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.193094    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q" event={"ID":"987395c3-26da-4d13-8564-f329160dde1d","Type":"ContainerDied","Data":"dfe915bcbfc27e690a8a535d1175a38ad6762cd9dc60b9d4682b04ea7f5a113d"}
Sep 09 16:29:22.193473 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.193035    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"
Sep 09 16:29:22.193473 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.193124    2576 scope.go:117] "RemoveContainer" containerID="4b274b3800974c703dffd3a4ac38bb2c0751e3e079ea9fc0860e5a36d59a35bb"
Sep 09 16:29:22.194607 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.194574    2576 generic.go:356] "Generic (PLEG): container finished" podID="8adb5336-488c-4988-93c8-6c160ab5889d" containerID="58820fd27f571e87a5a2796e6d4a79ee5b5118de19118cb0b2ad9f3ab861edeb" exitCode=0
Sep 09 16:29:22.194689 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.194635    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"
Sep 09 16:29:22.194689 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.194637    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j" event={"ID":"8adb5336-488c-4988-93c8-6c160ab5889d","Type":"ContainerDied","Data":"58820fd27f571e87a5a2796e6d4a79ee5b5118de19118cb0b2ad9f3ab861edeb"}
Sep 09 16:29:22.194689 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.194677    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j" event={"ID":"8adb5336-488c-4988-93c8-6c160ab5889d","Type":"ContainerDied","Data":"be87381012442e2fc0ff9fa827b5d3755a642e75e974593a6236a38b3d5b6a25"}
Sep 09 16:29:22.204253 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.204232    2576 scope.go:117] "RemoveContainer" containerID="4b274b3800974c703dffd3a4ac38bb2c0751e3e079ea9fc0860e5a36d59a35bb"
Sep 09 16:29:22.204609 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:29:22.204588    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b274b3800974c703dffd3a4ac38bb2c0751e3e079ea9fc0860e5a36d59a35bb\": container with ID starting with 4b274b3800974c703dffd3a4ac38bb2c0751e3e079ea9fc0860e5a36d59a35bb not found: ID does not exist" containerID="4b274b3800974c703dffd3a4ac38bb2c0751e3e079ea9fc0860e5a36d59a35bb"
Sep 09 16:29:22.204721 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.204622    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b274b3800974c703dffd3a4ac38bb2c0751e3e079ea9fc0860e5a36d59a35bb"} err="failed to get container status \"4b274b3800974c703dffd3a4ac38bb2c0751e3e079ea9fc0860e5a36d59a35bb\": rpc error: code = NotFound desc = could not find container \"4b274b3800974c703dffd3a4ac38bb2c0751e3e079ea9fc0860e5a36d59a35bb\": container with ID starting with 4b274b3800974c703dffd3a4ac38bb2c0751e3e079ea9fc0860e5a36d59a35bb not found: ID does not exist"
Sep 09 16:29:22.204721 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.204647    2576 scope.go:117] "RemoveContainer" containerID="58820fd27f571e87a5a2796e6d4a79ee5b5118de19118cb0b2ad9f3ab861edeb"
Sep 09 16:29:22.212518 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.212501    2576 scope.go:117] "RemoveContainer" containerID="58820fd27f571e87a5a2796e6d4a79ee5b5118de19118cb0b2ad9f3ab861edeb"
Sep 09 16:29:22.212787 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:29:22.212770    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58820fd27f571e87a5a2796e6d4a79ee5b5118de19118cb0b2ad9f3ab861edeb\": container with ID starting with 58820fd27f571e87a5a2796e6d4a79ee5b5118de19118cb0b2ad9f3ab861edeb not found: ID does not exist" containerID="58820fd27f571e87a5a2796e6d4a79ee5b5118de19118cb0b2ad9f3ab861edeb"
Sep 09 16:29:22.212864 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.212796    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58820fd27f571e87a5a2796e6d4a79ee5b5118de19118cb0b2ad9f3ab861edeb"} err="failed to get container status \"58820fd27f571e87a5a2796e6d4a79ee5b5118de19118cb0b2ad9f3ab861edeb\": rpc error: code = NotFound desc = could not find container \"58820fd27f571e87a5a2796e6d4a79ee5b5118de19118cb0b2ad9f3ab861edeb\": container with ID starting with 58820fd27f571e87a5a2796e6d4a79ee5b5118de19118cb0b2ad9f3ab861edeb not found: ID does not exist"
Sep 09 16:29:22.215450 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.215431    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"]
Sep 09 16:29:22.221421 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.221400    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c23332-84b747df98-b8z8q"]
Sep 09 16:29:22.231029 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.231011    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"]
Sep 09 16:29:22.235301 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.235282    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c23332-7cd489bc55-qsj8j"]
Sep 09 16:29:22.552938 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.552902    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8adb5336-488c-4988-93c8-6c160ab5889d" path="/var/lib/kubelet/pods/8adb5336-488c-4988-93c8-6c160ab5889d/volumes"
Sep 09 16:29:22.553260 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:22.553247    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="987395c3-26da-4d13-8564-f329160dde1d" path="/var/lib/kubelet/pods/987395c3-26da-4d13-8564-f329160dde1d/volumes"
Sep 09 16:29:30.181776 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:30.181747    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"
Sep 09 16:29:30.182164 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:30.181823    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"
Sep 09 16:29:33.559557 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.559521    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"]
Sep 09 16:29:33.559960 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.559827    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8adb5336-488c-4988-93c8-6c160ab5889d" containerName="payload-processing"
Sep 09 16:29:33.559960 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.559840    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8adb5336-488c-4988-93c8-6c160ab5889d" containerName="payload-processing"
Sep 09 16:29:33.559960 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.559850    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987395c3-26da-4d13-8564-f329160dde1d" containerName="payload-pre-processing"
Sep 09 16:29:33.559960 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.559856    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="987395c3-26da-4d13-8564-f329160dde1d" containerName="payload-pre-processing"
Sep 09 16:29:33.559960 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.559901    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8adb5336-488c-4988-93c8-6c160ab5889d" containerName="payload-processing"
Sep 09 16:29:33.559960 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.559909    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="987395c3-26da-4d13-8564-f329160dde1d" containerName="payload-pre-processing"
Sep 09 16:29:33.564636 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.564619    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"
Sep 09 16:29:33.566913 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.566890    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-041bf1\""
Sep 09 16:29:33.573543 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.573518    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"]
Sep 09 16:29:33.588722 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.588681    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"]
Sep 09 16:29:33.591807 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.591790    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"
Sep 09 16:29:33.594180 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.594165    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-041bf1-dockercfg-nm7jm\""
Sep 09 16:29:33.603558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.603538    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"]
Sep 09 16:29:33.660181 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.660157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d864c81-b058-493b-863a-3318edcdba59-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp\" (UID: \"8d864c81-b058-493b-863a-3318edcdba59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"
Sep 09 16:29:33.660272 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.660198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j79r\" (UniqueName: \"kubernetes.io/projected/8d864c81-b058-493b-863a-3318edcdba59-kube-api-access-8j79r\") pod \"payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp\" (UID: \"8d864c81-b058-493b-863a-3318edcdba59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"
Sep 09 16:29:33.761175 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.761146    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55rk\" (UniqueName: \"kubernetes.io/projected/eff2d442-febc-428f-92a4-1cd6690ce32a-kube-api-access-w55rk\") pod \"payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7\" (UID: \"eff2d442-febc-428f-92a4-1cd6690ce32a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"
Sep 09 16:29:33.761330 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.761184    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j79r\" (UniqueName: \"kubernetes.io/projected/8d864c81-b058-493b-863a-3318edcdba59-kube-api-access-8j79r\") pod \"payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp\" (UID: \"8d864c81-b058-493b-863a-3318edcdba59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"
Sep 09 16:29:33.761330 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.761211    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff2d442-febc-428f-92a4-1cd6690ce32a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7\" (UID: \"eff2d442-febc-428f-92a4-1cd6690ce32a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"
Sep 09 16:29:33.761401 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.761333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d864c81-b058-493b-863a-3318edcdba59-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp\" (UID: \"8d864c81-b058-493b-863a-3318edcdba59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"
Sep 09 16:29:33.761959 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.761940    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d864c81-b058-493b-863a-3318edcdba59-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp\" (UID: \"8d864c81-b058-493b-863a-3318edcdba59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"
Sep 09 16:29:33.769044 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.769025    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j79r\" (UniqueName: \"kubernetes.io/projected/8d864c81-b058-493b-863a-3318edcdba59-kube-api-access-8j79r\") pod \"payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp\" (UID: \"8d864c81-b058-493b-863a-3318edcdba59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"
Sep 09 16:29:33.862172 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.862103    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55rk\" (UniqueName: \"kubernetes.io/projected/eff2d442-febc-428f-92a4-1cd6690ce32a-kube-api-access-w55rk\") pod \"payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7\" (UID: \"eff2d442-febc-428f-92a4-1cd6690ce32a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"
Sep 09 16:29:33.862172 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.862144    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff2d442-febc-428f-92a4-1cd6690ce32a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7\" (UID: \"eff2d442-febc-428f-92a4-1cd6690ce32a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"
Sep 09 16:29:33.862834 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.862817    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff2d442-febc-428f-92a4-1cd6690ce32a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7\" (UID: \"eff2d442-febc-428f-92a4-1cd6690ce32a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"
Sep 09 16:29:33.869789 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.869766    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55rk\" (UniqueName: \"kubernetes.io/projected/eff2d442-febc-428f-92a4-1cd6690ce32a-kube-api-access-w55rk\") pod \"payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7\" (UID: \"eff2d442-febc-428f-92a4-1cd6690ce32a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"
Sep 09 16:29:33.875257 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.875232    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"
Sep 09 16:29:33.901236 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:33.901213    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"
Sep 09 16:29:34.149485 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:34.149464    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"]
Sep 09 16:29:34.151529 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:29:34.151492    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d864c81_b058_493b_863a_3318edcdba59.slice/crio-9f889ce67b6af59fa96b6359a363c7a880808f5a339b789c6d47e8aefb68a092 WatchSource:0}: Error finding container 9f889ce67b6af59fa96b6359a363c7a880808f5a339b789c6d47e8aefb68a092: Status 404 returned error can't find the container with id 9f889ce67b6af59fa96b6359a363c7a880808f5a339b789c6d47e8aefb68a092
Sep 09 16:29:34.180355 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:34.180319    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"]
Sep 09 16:29:34.182674 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:29:34.182649    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff2d442_febc_428f_92a4_1cd6690ce32a.slice/crio-442649054e77082b1a8a5e0ca017e3c7ae3cd16b9c9c0c54447a4e81c95f4f17 WatchSource:0}: Error finding container 442649054e77082b1a8a5e0ca017e3c7ae3cd16b9c9c0c54447a4e81c95f4f17: Status 404 returned error can't find the container with id 442649054e77082b1a8a5e0ca017e3c7ae3cd16b9c9c0c54447a4e81c95f4f17
Sep 09 16:29:34.251922 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:34.251882    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7" event={"ID":"eff2d442-febc-428f-92a4-1cd6690ce32a","Type":"ContainerStarted","Data":"273e75f6d6618abd70598b0a2f88b978cc993c3ff38794bfe9855ca008811e8e"}
Sep 09 16:29:34.252080 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:34.251969    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7" event={"ID":"eff2d442-febc-428f-92a4-1cd6690ce32a","Type":"ContainerStarted","Data":"442649054e77082b1a8a5e0ca017e3c7ae3cd16b9c9c0c54447a4e81c95f4f17"}
Sep 09 16:29:34.252080 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:34.252007    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"
Sep 09 16:29:34.253730 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:34.253682    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp" event={"ID":"8d864c81-b058-493b-863a-3318edcdba59","Type":"ContainerStarted","Data":"911f5f5fb1d32637a9698856a620cce0a87bce6c1e3d62f1da33d8791d31dd6b"}
Sep 09 16:29:34.253867 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:34.253753    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp" event={"ID":"8d864c81-b058-493b-863a-3318edcdba59","Type":"ContainerStarted","Data":"9f889ce67b6af59fa96b6359a363c7a880808f5a339b789c6d47e8aefb68a092"}
Sep 09 16:29:34.253867 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:34.253795    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"
Sep 09 16:29:34.271726 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:34.268003    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7" podStartSLOduration=1.267985916 podStartE2EDuration="1.267985916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:29:34.266789496 +0000 UTC m=+1250.202407930" watchObservedRunningTime="2026-09-09 16:29:34.267985916 +0000 UTC m=+1250.203604333"
Sep 09 16:29:34.283460 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:34.283403    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp" podStartSLOduration=1.283386473 podStartE2EDuration="1.283386473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:29:34.282034384 +0000 UTC m=+1250.217652835" watchObservedRunningTime="2026-09-09 16:29:34.283386473 +0000 UTC m=+1250.219004902"
Sep 09 16:29:45.260487 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:45.260455    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"
Sep 09 16:29:45.260897 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:45.260531    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"
Sep 09 16:29:48.909805 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:48.909768    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc"]
Sep 09 16:29:48.914521 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:48.914505    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc"
Sep 09 16:29:48.917094 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:48.917068    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:29:48.917214 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:48.917121    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxghs\""
Sep 09 16:29:48.920361 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:48.920336    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc"]
Sep 09 16:29:48.983036 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:48.982995    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c3dfcd7-0012-471b-bca7-4271d0cf3d35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-13b02364-sbfhc\" (UID: \"9c3dfcd7-0012-471b-bca7-4271d0cf3d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc"
Sep 09 16:29:49.084264 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:49.084227    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c3dfcd7-0012-471b-bca7-4271d0cf3d35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-13b02364-sbfhc\" (UID: \"9c3dfcd7-0012-471b-bca7-4271d0cf3d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc"
Sep 09 16:29:49.084845 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:49.084827    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c3dfcd7-0012-471b-bca7-4271d0cf3d35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-13b02364-sbfhc\" (UID: \"9c3dfcd7-0012-471b-bca7-4271d0cf3d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc"
Sep 09 16:29:49.225039 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:49.225011    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc"
Sep 09 16:29:49.505329 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:49.505302    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc"]
Sep 09 16:29:49.506726 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:29:49.506683    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3dfcd7_0012_471b_bca7_4271d0cf3d35.slice/crio-38101a8792644a3843f4711a503478c2c8c31045f70f242f0355f8898114b54d WatchSource:0}: Error finding container 38101a8792644a3843f4711a503478c2c8c31045f70f242f0355f8898114b54d: Status 404 returned error can't find the container with id 38101a8792644a3843f4711a503478c2c8c31045f70f242f0355f8898114b54d
Sep 09 16:29:49.508682 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:49.508664    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:29:50.326558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:50.326520    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc" event={"ID":"9c3dfcd7-0012-471b-bca7-4271d0cf3d35","Type":"ContainerStarted","Data":"38101a8792644a3843f4711a503478c2c8c31045f70f242f0355f8898114b54d"}
Sep 09 16:29:52.338925 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:52.338887    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc" event={"ID":"9c3dfcd7-0012-471b-bca7-4271d0cf3d35","Type":"ContainerStarted","Data":"830cfac61b0733bb6edc3a07102b06d4b6742968e3a30956f91763600f53ba9c"}
Sep 09 16:29:52.354728 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:52.354663    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc" podStartSLOduration=1.952399315 podStartE2EDuration="4.354648379s" totalImagesPullingTime="2.402249064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:29:52.353685932 +0000 UTC m=+1268.289304374" watchObservedRunningTime="2026-09-09 16:29:52.354648379 +0000 UTC m=+1268.290266794"
Sep 09 16:29:53.344352 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:53.344317    2576 generic.go:356] "Generic (PLEG): container finished" podID="9c3dfcd7-0012-471b-bca7-4271d0cf3d35" containerID="830cfac61b0733bb6edc3a07102b06d4b6742968e3a30956f91763600f53ba9c" exitCode=0
Sep 09 16:29:53.344732 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:53.344365    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc" event={"ID":"9c3dfcd7-0012-471b-bca7-4271d0cf3d35","Type":"ContainerDied","Data":"830cfac61b0733bb6edc3a07102b06d4b6742968e3a30956f91763600f53ba9c"}
Sep 09 16:29:54.474279 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:54.474254    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc"
Sep 09 16:29:54.530844 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:54.530809    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c3dfcd7-0012-471b-bca7-4271d0cf3d35-maas-api-service-ca\") pod \"9c3dfcd7-0012-471b-bca7-4271d0cf3d35\" (UID: \"9c3dfcd7-0012-471b-bca7-4271d0cf3d35\") "
Sep 09 16:29:54.531189 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:54.531164    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3dfcd7-0012-471b-bca7-4271d0cf3d35-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c3dfcd7-0012-471b-bca7-4271d0cf3d35" (UID: "9c3dfcd7-0012-471b-bca7-4271d0cf3d35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:29:54.631465 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:54.631389    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c3dfcd7-0012-471b-bca7-4271d0cf3d35-maas-api-service-ca\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:29:55.354298 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:55.354260    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc" event={"ID":"9c3dfcd7-0012-471b-bca7-4271d0cf3d35","Type":"ContainerDied","Data":"38101a8792644a3843f4711a503478c2c8c31045f70f242f0355f8898114b54d"}
Sep 09 16:29:55.354298 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:55.354284    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc"
Sep 09 16:29:55.354514 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:55.354317    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38101a8792644a3843f4711a503478c2c8c31045f70f242f0355f8898114b54d"
Sep 09 16:29:59.743613 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:59.743582    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"]
Sep 09 16:29:59.744062 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:59.743852    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q" podUID="67524555-833c-4f09-8fe6-b81fc49d648b" containerName="payload-processing" containerID="cri-o://0ce32297691839e86153429ecec3f8871f765231beaed0f47bd3fcf173e1ec93" gracePeriod=30
Sep 09 16:29:59.752894 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:59.752868    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"]
Sep 09 16:29:59.753133 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:59.753055    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg" podUID="5457463b-6eb5-4150-89a8-6981100c6530" containerName="payload-pre-processing" containerID="cri-o://44a5cf00a623ab5f0cddfd8a9fc3bcdf617b65d94fa21620a051d997a43f96e3" gracePeriod=30
Sep 09 16:29:59.901179 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:59.901144    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc"]
Sep 09 16:29:59.902567 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:29:59.902546    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-13b02364-sbfhc"]
Sep 09 16:30:00.018213 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.018176    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"
Sep 09 16:30:00.021229 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.021208    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"
Sep 09 16:30:00.076929 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.076903    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbbj4\" (UniqueName: \"kubernetes.io/projected/5457463b-6eb5-4150-89a8-6981100c6530-kube-api-access-qbbj4\") pod \"5457463b-6eb5-4150-89a8-6981100c6530\" (UID: \"5457463b-6eb5-4150-89a8-6981100c6530\") "
Sep 09 16:30:00.077116 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.076937    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67524555-833c-4f09-8fe6-b81fc49d648b-plugins-config-volume\") pod \"67524555-833c-4f09-8fe6-b81fc49d648b\" (UID: \"67524555-833c-4f09-8fe6-b81fc49d648b\") "
Sep 09 16:30:00.077116 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.076976    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s25nj\" (UniqueName: \"kubernetes.io/projected/67524555-833c-4f09-8fe6-b81fc49d648b-kube-api-access-s25nj\") pod \"67524555-833c-4f09-8fe6-b81fc49d648b\" (UID: \"67524555-833c-4f09-8fe6-b81fc49d648b\") "
Sep 09 16:30:00.077116 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.077042    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5457463b-6eb5-4150-89a8-6981100c6530-plugins-config-volume\") pod \"5457463b-6eb5-4150-89a8-6981100c6530\" (UID: \"5457463b-6eb5-4150-89a8-6981100c6530\") "
Sep 09 16:30:00.077436 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.077407    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67524555-833c-4f09-8fe6-b81fc49d648b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67524555-833c-4f09-8fe6-b81fc49d648b" (UID: "67524555-833c-4f09-8fe6-b81fc49d648b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:30:00.077436 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.077431    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5457463b-6eb5-4150-89a8-6981100c6530-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5457463b-6eb5-4150-89a8-6981100c6530" (UID: "5457463b-6eb5-4150-89a8-6981100c6530"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:30:00.079114 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.079091    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5457463b-6eb5-4150-89a8-6981100c6530-kube-api-access-qbbj4" (OuterVolumeSpecName: "kube-api-access-qbbj4") pod "5457463b-6eb5-4150-89a8-6981100c6530" (UID: "5457463b-6eb5-4150-89a8-6981100c6530"). InnerVolumeSpecName "kube-api-access-qbbj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:30:00.079215 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.079200    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67524555-833c-4f09-8fe6-b81fc49d648b-kube-api-access-s25nj" (OuterVolumeSpecName: "kube-api-access-s25nj") pod "67524555-833c-4f09-8fe6-b81fc49d648b" (UID: "67524555-833c-4f09-8fe6-b81fc49d648b"). InnerVolumeSpecName "kube-api-access-s25nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:30:00.132527 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.132503    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-041bf1-29816190-fw5kl"]
Sep 09 16:30:00.132809 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.132798    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67524555-833c-4f09-8fe6-b81fc49d648b" containerName="payload-processing"
Sep 09 16:30:00.132862 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.132811    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="67524555-833c-4f09-8fe6-b81fc49d648b" containerName="payload-processing"
Sep 09 16:30:00.132862 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.132830    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c3dfcd7-0012-471b-bca7-4271d0cf3d35" containerName="revoke-keys"
Sep 09 16:30:00.132862 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.132836    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c3dfcd7-0012-471b-bca7-4271d0cf3d35" containerName="revoke-keys"
Sep 09 16:30:00.132862 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.132844    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5457463b-6eb5-4150-89a8-6981100c6530" containerName="payload-pre-processing"
Sep 09 16:30:00.132862 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.132850    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5457463b-6eb5-4150-89a8-6981100c6530" containerName="payload-pre-processing"
Sep 09 16:30:00.133019 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.132896    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c3dfcd7-0012-471b-bca7-4271d0cf3d35" containerName="revoke-keys"
Sep 09 16:30:00.133019 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.132904    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="67524555-833c-4f09-8fe6-b81fc49d648b" containerName="payload-processing"
Sep 09 16:30:00.133019 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.132914    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5457463b-6eb5-4150-89a8-6981100c6530" containerName="payload-pre-processing"
Sep 09 16:30:00.135980 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.135962    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-041bf1-29816190-fw5kl"
Sep 09 16:30:00.138222 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.138205    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxghs\""
Sep 09 16:30:00.146215 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.146190    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-041bf1-29816190-fw5kl"
Sep 09 16:30:00.146559 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.146539    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-041bf1-29816190-fw5kl"]
Sep 09 16:30:00.178399 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.178371    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5457463b-6eb5-4150-89a8-6981100c6530-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:30:00.178399 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.178397    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbbj4\" (UniqueName: \"kubernetes.io/projected/5457463b-6eb5-4150-89a8-6981100c6530-kube-api-access-qbbj4\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:30:00.178560 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.178407    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67524555-833c-4f09-8fe6-b81fc49d648b-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:30:00.178560 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.178415    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s25nj\" (UniqueName: \"kubernetes.io/projected/67524555-833c-4f09-8fe6-b81fc49d648b-kube-api-access-s25nj\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:30:00.377814 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.377691    2576 generic.go:356] "Generic (PLEG): container finished" podID="5457463b-6eb5-4150-89a8-6981100c6530" containerID="44a5cf00a623ab5f0cddfd8a9fc3bcdf617b65d94fa21620a051d997a43f96e3" exitCode=0
Sep 09 16:30:00.377814 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.377746    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"
Sep 09 16:30:00.377814 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.377796    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg" event={"ID":"5457463b-6eb5-4150-89a8-6981100c6530","Type":"ContainerDied","Data":"44a5cf00a623ab5f0cddfd8a9fc3bcdf617b65d94fa21620a051d997a43f96e3"}
Sep 09 16:30:00.378091 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.377851    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg" event={"ID":"5457463b-6eb5-4150-89a8-6981100c6530","Type":"ContainerDied","Data":"b160ebfb248056441087ed43e8336b24454e0f9b15960fb7fea0b8c685a3f90a"}
Sep 09 16:30:00.378091 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.377883    2576 scope.go:117] "RemoveContainer" containerID="44a5cf00a623ab5f0cddfd8a9fc3bcdf617b65d94fa21620a051d997a43f96e3"
Sep 09 16:30:00.379200 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.379175    2576 generic.go:356] "Generic (PLEG): container finished" podID="67524555-833c-4f09-8fe6-b81fc49d648b" containerID="0ce32297691839e86153429ecec3f8871f765231beaed0f47bd3fcf173e1ec93" exitCode=0
Sep 09 16:30:00.379339 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.379217    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"
Sep 09 16:30:00.379339 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.379217    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q" event={"ID":"67524555-833c-4f09-8fe6-b81fc49d648b","Type":"ContainerDied","Data":"0ce32297691839e86153429ecec3f8871f765231beaed0f47bd3fcf173e1ec93"}
Sep 09 16:30:00.379339 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.379332    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q" event={"ID":"67524555-833c-4f09-8fe6-b81fc49d648b","Type":"ContainerDied","Data":"b61e22f94d47e061d29f22ce8fddc1965c978e8f5ec1c80dacc5a3e36f876270"}
Sep 09 16:30:00.387751 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.387735    2576 scope.go:117] "RemoveContainer" containerID="44a5cf00a623ab5f0cddfd8a9fc3bcdf617b65d94fa21620a051d997a43f96e3"
Sep 09 16:30:00.388060 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:30:00.388036    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a5cf00a623ab5f0cddfd8a9fc3bcdf617b65d94fa21620a051d997a43f96e3\": container with ID starting with 44a5cf00a623ab5f0cddfd8a9fc3bcdf617b65d94fa21620a051d997a43f96e3 not found: ID does not exist" containerID="44a5cf00a623ab5f0cddfd8a9fc3bcdf617b65d94fa21620a051d997a43f96e3"
Sep 09 16:30:00.388166 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.388071    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a5cf00a623ab5f0cddfd8a9fc3bcdf617b65d94fa21620a051d997a43f96e3"} err="failed to get container status \"44a5cf00a623ab5f0cddfd8a9fc3bcdf617b65d94fa21620a051d997a43f96e3\": rpc error: code = NotFound desc = could not find container \"44a5cf00a623ab5f0cddfd8a9fc3bcdf617b65d94fa21620a051d997a43f96e3\": container with ID starting with 44a5cf00a623ab5f0cddfd8a9fc3bcdf617b65d94fa21620a051d997a43f96e3 not found: ID does not exist"
Sep 09 16:30:00.388166 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.388089    2576 scope.go:117] "RemoveContainer" containerID="0ce32297691839e86153429ecec3f8871f765231beaed0f47bd3fcf173e1ec93"
Sep 09 16:30:00.395498 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.395475    2576 scope.go:117] "RemoveContainer" containerID="0ce32297691839e86153429ecec3f8871f765231beaed0f47bd3fcf173e1ec93"
Sep 09 16:30:00.395800 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:30:00.395782    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce32297691839e86153429ecec3f8871f765231beaed0f47bd3fcf173e1ec93\": container with ID starting with 0ce32297691839e86153429ecec3f8871f765231beaed0f47bd3fcf173e1ec93 not found: ID does not exist" containerID="0ce32297691839e86153429ecec3f8871f765231beaed0f47bd3fcf173e1ec93"
Sep 09 16:30:00.395902 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.395808    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce32297691839e86153429ecec3f8871f765231beaed0f47bd3fcf173e1ec93"} err="failed to get container status \"0ce32297691839e86153429ecec3f8871f765231beaed0f47bd3fcf173e1ec93\": rpc error: code = NotFound desc = could not find container \"0ce32297691839e86153429ecec3f8871f765231beaed0f47bd3fcf173e1ec93\": container with ID starting with 0ce32297691839e86153429ecec3f8871f765231beaed0f47bd3fcf173e1ec93 not found: ID does not exist"
Sep 09 16:30:00.399152 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.399128    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"]
Sep 09 16:30:00.401504 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.401483    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-13b02364-5c47789f74-4vxxg"]
Sep 09 16:30:00.411256 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.411233    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"]
Sep 09 16:30:00.414823 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.414805    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-041bf1-29816190-fw5kl"]
Sep 09 16:30:00.422866 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.422838    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-13b02364-d4d456597-7dq7q"]
Sep 09 16:30:00.552763 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.552728    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5457463b-6eb5-4150-89a8-6981100c6530" path="/var/lib/kubelet/pods/5457463b-6eb5-4150-89a8-6981100c6530/volumes"
Sep 09 16:30:00.553100 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.553087    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67524555-833c-4f09-8fe6-b81fc49d648b" path="/var/lib/kubelet/pods/67524555-833c-4f09-8fe6-b81fc49d648b/volumes"
Sep 09 16:30:00.553387 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:00.553376    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c3dfcd7-0012-471b-bca7-4271d0cf3d35" path="/var/lib/kubelet/pods/9c3dfcd7-0012-471b-bca7-4271d0cf3d35/volumes"
Sep 09 16:30:01.387063 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:01.387034    2576 generic.go:356] "Generic (PLEG): container finished" podID="9b58f933-3179-45a8-ac10-a53faa2690a4" containerID="d00bf8fbf9ef4a2922d08cb762392b290fa34b9c15efd7db33d26925ab61dd40" exitCode=0
Sep 09 16:30:01.387417 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:01.387078    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-041bf1-29816190-fw5kl" event={"ID":"9b58f933-3179-45a8-ac10-a53faa2690a4","Type":"ContainerDied","Data":"d00bf8fbf9ef4a2922d08cb762392b290fa34b9c15efd7db33d26925ab61dd40"}
Sep 09 16:30:01.387417 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:01.387127    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-041bf1-29816190-fw5kl" event={"ID":"9b58f933-3179-45a8-ac10-a53faa2690a4","Type":"ContainerStarted","Data":"ba10ba6bdd7d57feca010264a3d12a5846ca201634a0c8dbf54fa9d14d26e314"}
Sep 09 16:30:02.257515 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.257481    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"]
Sep 09 16:30:02.260857 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.260842    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"
Sep 09 16:30:02.263347 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.263329    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-9520a4-dockercfg-2h596\""
Sep 09 16:30:02.263460 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.263329    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-9520a4\""
Sep 09 16:30:02.270316 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.270291    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"]
Sep 09 16:30:02.293328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.293285    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"]
Sep 09 16:30:02.296608 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.296593    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"
Sep 09 16:30:02.309900 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.309875    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"]
Sep 09 16:30:02.394426 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.394404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2kkv\" (UniqueName: \"kubernetes.io/projected/6b4187cd-3eeb-47c4-b885-aa26f2b9d306-kube-api-access-r2kkv\") pod \"payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm\" (UID: \"6b4187cd-3eeb-47c4-b885-aa26f2b9d306\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"
Sep 09 16:30:02.394733 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.394438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbmkz\" (UniqueName: \"kubernetes.io/projected/8ea89d17-f59c-413d-848f-0bf035f59da7-kube-api-access-hbmkz\") pod \"payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5\" (UID: \"8ea89d17-f59c-413d-848f-0bf035f59da7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"
Sep 09 16:30:02.394733 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.394502    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4187cd-3eeb-47c4-b885-aa26f2b9d306-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm\" (UID: \"6b4187cd-3eeb-47c4-b885-aa26f2b9d306\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"
Sep 09 16:30:02.394733 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.394548    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea89d17-f59c-413d-848f-0bf035f59da7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5\" (UID: \"8ea89d17-f59c-413d-848f-0bf035f59da7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"
Sep 09 16:30:02.495419 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.495386    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2kkv\" (UniqueName: \"kubernetes.io/projected/6b4187cd-3eeb-47c4-b885-aa26f2b9d306-kube-api-access-r2kkv\") pod \"payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm\" (UID: \"6b4187cd-3eeb-47c4-b885-aa26f2b9d306\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"
Sep 09 16:30:02.495598 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.495428    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbmkz\" (UniqueName: \"kubernetes.io/projected/8ea89d17-f59c-413d-848f-0bf035f59da7-kube-api-access-hbmkz\") pod \"payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5\" (UID: \"8ea89d17-f59c-413d-848f-0bf035f59da7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"
Sep 09 16:30:02.495598 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.495457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4187cd-3eeb-47c4-b885-aa26f2b9d306-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm\" (UID: \"6b4187cd-3eeb-47c4-b885-aa26f2b9d306\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"
Sep 09 16:30:02.495598 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.495480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea89d17-f59c-413d-848f-0bf035f59da7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5\" (UID: \"8ea89d17-f59c-413d-848f-0bf035f59da7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"
Sep 09 16:30:02.496312 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.496287    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea89d17-f59c-413d-848f-0bf035f59da7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5\" (UID: \"8ea89d17-f59c-413d-848f-0bf035f59da7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"
Sep 09 16:30:02.496394 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.496312    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4187cd-3eeb-47c4-b885-aa26f2b9d306-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm\" (UID: \"6b4187cd-3eeb-47c4-b885-aa26f2b9d306\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"
Sep 09 16:30:02.503946 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.503916    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbmkz\" (UniqueName: \"kubernetes.io/projected/8ea89d17-f59c-413d-848f-0bf035f59da7-kube-api-access-hbmkz\") pod \"payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5\" (UID: \"8ea89d17-f59c-413d-848f-0bf035f59da7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"
Sep 09 16:30:02.504046 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.503958    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2kkv\" (UniqueName: \"kubernetes.io/projected/6b4187cd-3eeb-47c4-b885-aa26f2b9d306-kube-api-access-r2kkv\") pod \"payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm\" (UID: \"6b4187cd-3eeb-47c4-b885-aa26f2b9d306\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"
Sep 09 16:30:02.519988 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.519935    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-041bf1-29816190-fw5kl"
Sep 09 16:30:02.572846 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.572821    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"
Sep 09 16:30:02.606684 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.606654    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"
Sep 09 16:30:02.842453 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.842427    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"]
Sep 09 16:30:02.847631 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:30:02.847601    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4187cd_3eeb_47c4_b885_aa26f2b9d306.slice/crio-8688bb43fd820d883be0584c277aa22c501c48f21dfdb076ca8956b239131340 WatchSource:0}: Error finding container 8688bb43fd820d883be0584c277aa22c501c48f21dfdb076ca8956b239131340: Status 404 returned error can't find the container with id 8688bb43fd820d883be0584c277aa22c501c48f21dfdb076ca8956b239131340
Sep 09 16:30:02.887891 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:02.887868    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"]
Sep 09 16:30:02.891021 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:30:02.890995    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea89d17_f59c_413d_848f_0bf035f59da7.slice/crio-30d4d8725368f2ef2c1c2d1dbc019f70d2b20979659c2bc78bd9eccf3a31f0ec WatchSource:0}: Error finding container 30d4d8725368f2ef2c1c2d1dbc019f70d2b20979659c2bc78bd9eccf3a31f0ec: Status 404 returned error can't find the container with id 30d4d8725368f2ef2c1c2d1dbc019f70d2b20979659c2bc78bd9eccf3a31f0ec
Sep 09 16:30:03.398625 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:03.398587    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm" event={"ID":"6b4187cd-3eeb-47c4-b885-aa26f2b9d306","Type":"ContainerStarted","Data":"b23276beee4f0421fc4969d2a88f2efecc05ae7baea0ca65ed90a26f2a15061a"}
Sep 09 16:30:03.399131 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:03.398657    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm" event={"ID":"6b4187cd-3eeb-47c4-b885-aa26f2b9d306","Type":"ContainerStarted","Data":"8688bb43fd820d883be0584c277aa22c501c48f21dfdb076ca8956b239131340"}
Sep 09 16:30:03.399131 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:03.398810    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"
Sep 09 16:30:03.400327 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:03.400297    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5" event={"ID":"8ea89d17-f59c-413d-848f-0bf035f59da7","Type":"ContainerStarted","Data":"21000a445eaac40e0350ac38beea33a01196287051f1d689680c578ed46155ee"}
Sep 09 16:30:03.400462 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:03.400337    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5" event={"ID":"8ea89d17-f59c-413d-848f-0bf035f59da7","Type":"ContainerStarted","Data":"30d4d8725368f2ef2c1c2d1dbc019f70d2b20979659c2bc78bd9eccf3a31f0ec"}
Sep 09 16:30:03.400462 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:03.400383    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"
Sep 09 16:30:03.401681 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:03.401654    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-041bf1-29816190-fw5kl"
Sep 09 16:30:03.401782 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:03.401652    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-041bf1-29816190-fw5kl" event={"ID":"9b58f933-3179-45a8-ac10-a53faa2690a4","Type":"ContainerDied","Data":"ba10ba6bdd7d57feca010264a3d12a5846ca201634a0c8dbf54fa9d14d26e314"}
Sep 09 16:30:03.401782 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:03.401749    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba10ba6bdd7d57feca010264a3d12a5846ca201634a0c8dbf54fa9d14d26e314"
Sep 09 16:30:03.415827 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:03.415772    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm" podStartSLOduration=1.415755814 podStartE2EDuration="1.415755814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:30:03.4140698 +0000 UTC m=+1279.349688236" watchObservedRunningTime="2026-09-09 16:30:03.415755814 +0000 UTC m=+1279.351374232"
Sep 09 16:30:03.434787 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:03.434740    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5" podStartSLOduration=1.434724885 podStartE2EDuration="1.434724885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:30:03.433539307 +0000 UTC m=+1279.369157748" watchObservedRunningTime="2026-09-09 16:30:03.434724885 +0000 UTC m=+1279.370343311"
Sep 09 16:30:14.408122 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:14.408092    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"
Sep 09 16:30:14.408557 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:14.408183    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"
Sep 09 16:30:24.801498 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.801467    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"]
Sep 09 16:30:24.802031 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.801879    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b58f933-3179-45a8-ac10-a53faa2690a4" containerName="cleanup"
Sep 09 16:30:24.802031 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.801896    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b58f933-3179-45a8-ac10-a53faa2690a4" containerName="cleanup"
Sep 09 16:30:24.802031 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.801964    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b58f933-3179-45a8-ac10-a53faa2690a4" containerName="cleanup"
Sep 09 16:30:24.806408 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.806388    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"
Sep 09 16:30:24.808564 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.808545    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-042f8b\""
Sep 09 16:30:24.815365 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.815341    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"]
Sep 09 16:30:24.828469 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.828451    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"]
Sep 09 16:30:24.830647 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.830631    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"
Sep 09 16:30:24.832840 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.832822    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-042f8b-dockercfg-hq8vc\""
Sep 09 16:30:24.842368 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.842350    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"]
Sep 09 16:30:24.986226 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.986181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l4q6\" (UniqueName: \"kubernetes.io/projected/eb09db09-f69c-4dad-a7dc-4ce7fe732674-kube-api-access-7l4q6\") pod \"payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl\" (UID: \"eb09db09-f69c-4dad-a7dc-4ce7fe732674\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"
Sep 09 16:30:24.986226 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.986229    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vgxw\" (UniqueName: \"kubernetes.io/projected/b4dd7c8c-f218-4f30-b6b6-087f4a1fa106-kube-api-access-6vgxw\") pod \"payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6\" (UID: \"b4dd7c8c-f218-4f30-b6b6-087f4a1fa106\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"
Sep 09 16:30:24.986431 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.986262    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb09db09-f69c-4dad-a7dc-4ce7fe732674-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl\" (UID: \"eb09db09-f69c-4dad-a7dc-4ce7fe732674\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"
Sep 09 16:30:24.986431 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:24.986277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4dd7c8c-f218-4f30-b6b6-087f4a1fa106-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6\" (UID: \"b4dd7c8c-f218-4f30-b6b6-087f4a1fa106\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"
Sep 09 16:30:25.087478 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.087404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l4q6\" (UniqueName: \"kubernetes.io/projected/eb09db09-f69c-4dad-a7dc-4ce7fe732674-kube-api-access-7l4q6\") pod \"payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl\" (UID: \"eb09db09-f69c-4dad-a7dc-4ce7fe732674\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"
Sep 09 16:30:25.087478 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.087440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vgxw\" (UniqueName: \"kubernetes.io/projected/b4dd7c8c-f218-4f30-b6b6-087f4a1fa106-kube-api-access-6vgxw\") pod \"payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6\" (UID: \"b4dd7c8c-f218-4f30-b6b6-087f4a1fa106\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"
Sep 09 16:30:25.087478 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.087471    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb09db09-f69c-4dad-a7dc-4ce7fe732674-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl\" (UID: \"eb09db09-f69c-4dad-a7dc-4ce7fe732674\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"
Sep 09 16:30:25.087829 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.087488    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4dd7c8c-f218-4f30-b6b6-087f4a1fa106-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6\" (UID: \"b4dd7c8c-f218-4f30-b6b6-087f4a1fa106\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"
Sep 09 16:30:25.088180 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.088157    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4dd7c8c-f218-4f30-b6b6-087f4a1fa106-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6\" (UID: \"b4dd7c8c-f218-4f30-b6b6-087f4a1fa106\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"
Sep 09 16:30:25.088261 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.088226    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb09db09-f69c-4dad-a7dc-4ce7fe732674-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl\" (UID: \"eb09db09-f69c-4dad-a7dc-4ce7fe732674\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"
Sep 09 16:30:25.095405 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.095383    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l4q6\" (UniqueName: \"kubernetes.io/projected/eb09db09-f69c-4dad-a7dc-4ce7fe732674-kube-api-access-7l4q6\") pod \"payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl\" (UID: \"eb09db09-f69c-4dad-a7dc-4ce7fe732674\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"
Sep 09 16:30:25.095524 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.095506    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vgxw\" (UniqueName: \"kubernetes.io/projected/b4dd7c8c-f218-4f30-b6b6-087f4a1fa106-kube-api-access-6vgxw\") pod \"payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6\" (UID: \"b4dd7c8c-f218-4f30-b6b6-087f4a1fa106\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"
Sep 09 16:30:25.118409 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.118387    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"
Sep 09 16:30:25.139937 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.139910    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"
Sep 09 16:30:25.390668 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.390492    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"]
Sep 09 16:30:25.394412 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:30:25.394384    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4dd7c8c_f218_4f30_b6b6_087f4a1fa106.slice/crio-4317e20ed29d5b56d5ee5aaa0e41fe5deaed8f2378d63713a4292df24926a82b WatchSource:0}: Error finding container 4317e20ed29d5b56d5ee5aaa0e41fe5deaed8f2378d63713a4292df24926a82b: Status 404 returned error can't find the container with id 4317e20ed29d5b56d5ee5aaa0e41fe5deaed8f2378d63713a4292df24926a82b
Sep 09 16:30:25.421853 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.421821    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"]
Sep 09 16:30:25.426820 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:30:25.426792    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb09db09_f69c_4dad_a7dc_4ce7fe732674.slice/crio-e5ef886e082226397a4faeb1e25155f4f551c242db5d019cb59095ea701055a6 WatchSource:0}: Error finding container e5ef886e082226397a4faeb1e25155f4f551c242db5d019cb59095ea701055a6: Status 404 returned error can't find the container with id e5ef886e082226397a4faeb1e25155f4f551c242db5d019cb59095ea701055a6
Sep 09 16:30:25.502558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.502519    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6" event={"ID":"b4dd7c8c-f218-4f30-b6b6-087f4a1fa106","Type":"ContainerStarted","Data":"1963bbbaad78dceb80d0d6d493a4ead0a6ab7653f7e87a03e4e0cba317e28623"}
Sep 09 16:30:25.502756 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.502594    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"
Sep 09 16:30:25.502756 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.502615    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6" event={"ID":"b4dd7c8c-f218-4f30-b6b6-087f4a1fa106","Type":"ContainerStarted","Data":"4317e20ed29d5b56d5ee5aaa0e41fe5deaed8f2378d63713a4292df24926a82b"}
Sep 09 16:30:25.504225 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.504191    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl" event={"ID":"eb09db09-f69c-4dad-a7dc-4ce7fe732674","Type":"ContainerStarted","Data":"a6b091b86021cc4ba5afb643f5302e7ff751ffe12d85e4a9b3db35a703055bef"}
Sep 09 16:30:25.504322 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.504248    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl" event={"ID":"eb09db09-f69c-4dad-a7dc-4ce7fe732674","Type":"ContainerStarted","Data":"e5ef886e082226397a4faeb1e25155f4f551c242db5d019cb59095ea701055a6"}
Sep 09 16:30:25.504322 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.504290    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"
Sep 09 16:30:25.518508 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.518455    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6" podStartSLOduration=1.518436773 podStartE2EDuration="1.518436773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:30:25.517590099 +0000 UTC m=+1301.453208528" watchObservedRunningTime="2026-09-09 16:30:25.518436773 +0000 UTC m=+1301.454055190"
Sep 09 16:30:25.534999 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:25.534951    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl" podStartSLOduration=1.5349351900000001 podStartE2EDuration="1.53493519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:30:25.534169632 +0000 UTC m=+1301.469788076" watchObservedRunningTime="2026-09-09 16:30:25.53493519 +0000 UTC m=+1301.470553605"
Sep 09 16:30:36.511115 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:36.511084    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"
Sep 09 16:30:36.511537 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:36.511149    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"
Sep 09 16:30:47.509336 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.509305    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"]
Sep 09 16:30:47.513388 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.513367    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.516755 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.516732    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-041bf1\"/\"kube-root-ca.crt\""
Sep 09 16:30:47.516914 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.516778    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-041bf1\"/\"openshift-service-ca.crt\""
Sep 09 16:30:47.516914 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.516804    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-041bf1\"/\"auth-test-model-041bf1-kserve-self-signed-certs\""
Sep 09 16:30:47.517039 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.516923    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-041bf1\"/\"default-dockercfg-gqlb2\""
Sep 09 16:30:47.522575 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.522551    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"]
Sep 09 16:30:47.563437 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.563413    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb41bc3-529d-45c9-ad64-83a36dea7500-tls-certs\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.563552 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.563450    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-model-cache\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.563552 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.563478    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dswh\" (UniqueName: \"kubernetes.io/projected/feb41bc3-529d-45c9-ad64-83a36dea7500-kube-api-access-5dswh\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.563552 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.563504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-home\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.563680 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.563547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-tmp-dir\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.563680 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.563633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-dshm\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.664205 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.664173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-dshm\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.664388 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.664226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb41bc3-529d-45c9-ad64-83a36dea7500-tls-certs\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.664388 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.664349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-model-cache\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.664505 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.664393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dswh\" (UniqueName: \"kubernetes.io/projected/feb41bc3-529d-45c9-ad64-83a36dea7500-kube-api-access-5dswh\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.664505 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.664423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-home\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.664505 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.664454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-tmp-dir\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.664787 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.664731    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-model-cache\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.664787 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.664771    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-home\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.664924 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.664860    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-tmp-dir\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.667093 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.667072    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-dshm\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.667452 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.667433    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb41bc3-529d-45c9-ad64-83a36dea7500-tls-certs\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.672143 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.672124    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dswh\" (UniqueName: \"kubernetes.io/projected/feb41bc3-529d-45c9-ad64-83a36dea7500-kube-api-access-5dswh\") pod \"auth-test-model-041bf1-kserve-8c48fc67f-pzvl8\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:47.823076 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:47.822993    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:48.091092 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:48.091022    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"]
Sep 09 16:30:48.093941 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:30:48.093910    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfeb41bc3_529d_45c9_ad64_83a36dea7500.slice/crio-cddaa6bc1a9da60bdef915311bf590cbdcd9eb82acd206de0f1f9778fde153c9 WatchSource:0}: Error finding container cddaa6bc1a9da60bdef915311bf590cbdcd9eb82acd206de0f1f9778fde153c9: Status 404 returned error can't find the container with id cddaa6bc1a9da60bdef915311bf590cbdcd9eb82acd206de0f1f9778fde153c9
Sep 09 16:30:48.611793 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:48.611748    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8" event={"ID":"feb41bc3-529d-45c9-ad64-83a36dea7500","Type":"ContainerStarted","Data":"6668012ee5c6724f3e94bfa9090a3cee85f6a62037a3de0d360f6e3986b02d80"}
Sep 09 16:30:48.611793 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:48.611798    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8" event={"ID":"feb41bc3-529d-45c9-ad64-83a36dea7500","Type":"ContainerStarted","Data":"cddaa6bc1a9da60bdef915311bf590cbdcd9eb82acd206de0f1f9778fde153c9"}
Sep 09 16:30:48.629055 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:48.629005    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8" podStartSLOduration=1.6289886679999999 podStartE2EDuration="1.628988668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:30:48.6279976 +0000 UTC m=+1324.563616015" watchObservedRunningTime="2026-09-09 16:30:48.628988668 +0000 UTC m=+1324.564607125"
Sep 09 16:30:57.823293 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:57.823264    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:57.823773 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:57.823334    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:57.835930 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:57.835908    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:30:58.667316 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:30:58.667289    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:31:17.549538 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:17.549464    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7685-139728178898528"]
Sep 09 16:31:17.551590 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:17.551574    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7685-139728178898528"
Sep 09 16:31:17.554035 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:17.554011    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bzk7m\""
Sep 09 16:31:17.556229 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:17.556205    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7685-139728178898528"]
Sep 09 16:31:17.726057 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:17.726025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghww5\" (UniqueName: \"kubernetes.io/projected/26473dae-efeb-4d77-9ce4-a41be68c638a-kube-api-access-ghww5\") pod \"test-curl-7685-139728178898528\" (UID: \"26473dae-efeb-4d77-9ce4-a41be68c638a\") " pod="opendatahub/test-curl-7685-139728178898528"
Sep 09 16:31:17.826612 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:17.826524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghww5\" (UniqueName: \"kubernetes.io/projected/26473dae-efeb-4d77-9ce4-a41be68c638a-kube-api-access-ghww5\") pod \"test-curl-7685-139728178898528\" (UID: \"26473dae-efeb-4d77-9ce4-a41be68c638a\") " pod="opendatahub/test-curl-7685-139728178898528"
Sep 09 16:31:17.835065 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:17.835032    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghww5\" (UniqueName: \"kubernetes.io/projected/26473dae-efeb-4d77-9ce4-a41be68c638a-kube-api-access-ghww5\") pod \"test-curl-7685-139728178898528\" (UID: \"26473dae-efeb-4d77-9ce4-a41be68c638a\") " pod="opendatahub/test-curl-7685-139728178898528"
Sep 09 16:31:17.861824 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:17.861804    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7685-139728178898528"
Sep 09 16:31:18.129512 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:18.129433    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7685-139728178898528"]
Sep 09 16:31:18.132351 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:31:18.132324    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26473dae_efeb_4d77_9ce4_a41be68c638a.slice/crio-55fa3d1ef93a6cc3411092029426746fcf91273a9f0bacc7714d54a08583db85 WatchSource:0}: Error finding container 55fa3d1ef93a6cc3411092029426746fcf91273a9f0bacc7714d54a08583db85: Status 404 returned error can't find the container with id 55fa3d1ef93a6cc3411092029426746fcf91273a9f0bacc7714d54a08583db85
Sep 09 16:31:18.742681 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:18.742645    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7685-139728178898528" event={"ID":"26473dae-efeb-4d77-9ce4-a41be68c638a","Type":"ContainerStarted","Data":"55fa3d1ef93a6cc3411092029426746fcf91273a9f0bacc7714d54a08583db85"}
Sep 09 16:31:21.506890 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.506862    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"]
Sep 09 16:31:21.509144 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.509129    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.512515 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.512495    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9520a4\"/\"openshift-service-ca.crt\""
Sep 09 16:31:21.512638 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.512522    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9520a4\"/\"auth-test-model-9520a4-kserve-self-signed-certs\""
Sep 09 16:31:21.512735 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.512716    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9520a4\"/\"default-dockercfg-qlph4\""
Sep 09 16:31:21.512893 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.512873    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9520a4\"/\"kube-root-ca.crt\""
Sep 09 16:31:21.518103 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.518082    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"]
Sep 09 16:31:21.661887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.661860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeffab74-534d-4cb4-9af1-40a5d38eb536-tls-certs\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.662047 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.661897    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-home\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.662047 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.661931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-tmp-dir\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.662047 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.662013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-dshm\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.662047 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.662039    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-model-cache\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.662184 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.662078    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb2gp\" (UniqueName: \"kubernetes.io/projected/eeffab74-534d-4cb4-9af1-40a5d38eb536-kube-api-access-zb2gp\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.758510 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.758422    2576 generic.go:356] "Generic (PLEG): container finished" podID="26473dae-efeb-4d77-9ce4-a41be68c638a" containerID="924af2be4c68138f9d3fe8a281f2a4ce995451f4e1474c1b2fcd1be22b707c5d" exitCode=0
Sep 09 16:31:21.758510 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.758463    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7685-139728178898528" event={"ID":"26473dae-efeb-4d77-9ce4-a41be68c638a","Type":"ContainerDied","Data":"924af2be4c68138f9d3fe8a281f2a4ce995451f4e1474c1b2fcd1be22b707c5d"}
Sep 09 16:31:21.762789 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.762767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-dshm\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.762901 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.762799    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-model-cache\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.762901 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.762853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb2gp\" (UniqueName: \"kubernetes.io/projected/eeffab74-534d-4cb4-9af1-40a5d38eb536-kube-api-access-zb2gp\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.762901 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.762893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeffab74-534d-4cb4-9af1-40a5d38eb536-tls-certs\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.763070 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.763028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-home\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.763134 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.763081    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-tmp-dir\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.763219 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.763194    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-model-cache\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.763329 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.763311    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-home\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.763422 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.763406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-tmp-dir\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.765153 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.765134    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-dshm\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.765398 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.765382    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeffab74-534d-4cb4-9af1-40a5d38eb536-tls-certs\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.776440 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.776416    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb2gp\" (UniqueName: \"kubernetes.io/projected/eeffab74-534d-4cb4-9af1-40a5d38eb536-kube-api-access-zb2gp\") pod \"auth-test-model-9520a4-kserve-7b4b94958c-kdpvl\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:21.818452 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:21.818430    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:22.088780 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:22.088757    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"]
Sep 09 16:31:22.091445 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:31:22.091416    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeffab74_534d_4cb4_9af1_40a5d38eb536.slice/crio-947c22ece67c8ec9e38159446c34d9aeacf416f169b78e538e4b9a23e0370ed3 WatchSource:0}: Error finding container 947c22ece67c8ec9e38159446c34d9aeacf416f169b78e538e4b9a23e0370ed3: Status 404 returned error can't find the container with id 947c22ece67c8ec9e38159446c34d9aeacf416f169b78e538e4b9a23e0370ed3
Sep 09 16:31:22.770639 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:22.770600    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl" event={"ID":"eeffab74-534d-4cb4-9af1-40a5d38eb536","Type":"ContainerStarted","Data":"209fc2d5b004581b91fb2725df9245f7f434a885f5984038df6c6fe4e75f2a1e"}
Sep 09 16:31:22.770639 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:22.770648    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl" event={"ID":"eeffab74-534d-4cb4-9af1-40a5d38eb536","Type":"ContainerStarted","Data":"947c22ece67c8ec9e38159446c34d9aeacf416f169b78e538e4b9a23e0370ed3"}
Sep 09 16:31:22.786945 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:22.786899    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl" podStartSLOduration=1.786883321 podStartE2EDuration="1.786883321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:31:22.785497873 +0000 UTC m=+1358.721116333" watchObservedRunningTime="2026-09-09 16:31:22.786883321 +0000 UTC m=+1358.722501737"
Sep 09 16:31:22.902530 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:22.902508    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7685-139728178898528"
Sep 09 16:31:22.937816 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:22.937796    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7685-139728178898528_26473dae-efeb-4d77-9ce4-a41be68c638a/test-curl-7685-139728178898528/0.log"
Sep 09 16:31:22.962160 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:22.962133    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7685-139728178898528"]
Sep 09 16:31:22.965919 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:22.965899    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7685-139728178898528"]
Sep 09 16:31:23.076925 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:23.076865    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghww5\" (UniqueName: \"kubernetes.io/projected/26473dae-efeb-4d77-9ce4-a41be68c638a-kube-api-access-ghww5\") pod \"26473dae-efeb-4d77-9ce4-a41be68c638a\" (UID: \"26473dae-efeb-4d77-9ce4-a41be68c638a\") "
Sep 09 16:31:23.079051 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:23.079026    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26473dae-efeb-4d77-9ce4-a41be68c638a-kube-api-access-ghww5" (OuterVolumeSpecName: "kube-api-access-ghww5") pod "26473dae-efeb-4d77-9ce4-a41be68c638a" (UID: "26473dae-efeb-4d77-9ce4-a41be68c638a"). InnerVolumeSpecName "kube-api-access-ghww5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:31:23.178193 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:23.178165    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghww5\" (UniqueName: \"kubernetes.io/projected/26473dae-efeb-4d77-9ce4-a41be68c638a-kube-api-access-ghww5\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:31:23.776488 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:23.776463    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55fa3d1ef93a6cc3411092029426746fcf91273a9f0bacc7714d54a08583db85"
Sep 09 16:31:23.776488 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:23.776464    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7685-139728178898528"
Sep 09 16:31:23.786093 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:31:23.786063    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7685-139728178898528\" is forbidden: User \"system:node:ip-10-0-137-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-117.ec2.internal' and this object" podUID="26473dae-efeb-4d77-9ce4-a41be68c638a" pod="opendatahub/test-curl-7685-139728178898528"
Sep 09 16:31:24.552042 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:31:24.551995    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7685-139728178898528\" is forbidden: User \"system:node:ip-10-0-137-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-117.ec2.internal' and this object" podUID="26473dae-efeb-4d77-9ce4-a41be68c638a" pod="opendatahub/test-curl-7685-139728178898528"
Sep 09 16:31:24.553478 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:24.553457    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26473dae-efeb-4d77-9ce4-a41be68c638a" path="/var/lib/kubelet/pods/26473dae-efeb-4d77-9ce4-a41be68c638a/volumes"
Sep 09 16:31:31.818618 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:31.818586    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:31.819163 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:31.818631    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:31.831036 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:31.831009    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:32.833413 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:32.833380    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:31:38.987779 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:38.987742    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"]
Sep 09 16:31:38.988137 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:38.988082    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26473dae-efeb-4d77-9ce4-a41be68c638a" containerName="test-curl-7685-139728178898528"
Sep 09 16:31:38.988137 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:38.988093    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="26473dae-efeb-4d77-9ce4-a41be68c638a" containerName="test-curl-7685-139728178898528"
Sep 09 16:31:38.988214 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:38.988160    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="26473dae-efeb-4d77-9ce4-a41be68c638a" containerName="test-curl-7685-139728178898528"
Sep 09 16:31:38.991301 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:38.991285    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"
Sep 09 16:31:38.993733 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:38.993689    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a0d4fa04-dockercfg-9x6f9\""
Sep 09 16:31:38.993733 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:38.993689    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a0d4fa04\""
Sep 09 16:31:39.001078 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:39.001050    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"]
Sep 09 16:31:39.114065 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:39.114036    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpmxg\" (UniqueName: \"kubernetes.io/projected/8ee6272d-838e-4ff6-9e03-7ce29638a088-kube-api-access-cpmxg\") pod \"payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp\" (UID: \"8ee6272d-838e-4ff6-9e03-7ce29638a088\") " pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"
Sep 09 16:31:39.114251 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:39.114091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee6272d-838e-4ff6-9e03-7ce29638a088-plugins-config-volume\") pod \"payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp\" (UID: \"8ee6272d-838e-4ff6-9e03-7ce29638a088\") " pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"
Sep 09 16:31:39.214803 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:39.214771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpmxg\" (UniqueName: \"kubernetes.io/projected/8ee6272d-838e-4ff6-9e03-7ce29638a088-kube-api-access-cpmxg\") pod \"payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp\" (UID: \"8ee6272d-838e-4ff6-9e03-7ce29638a088\") " pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"
Sep 09 16:31:39.214976 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:39.214877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee6272d-838e-4ff6-9e03-7ce29638a088-plugins-config-volume\") pod \"payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp\" (UID: \"8ee6272d-838e-4ff6-9e03-7ce29638a088\") " pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"
Sep 09 16:31:39.215614 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:39.215594    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee6272d-838e-4ff6-9e03-7ce29638a088-plugins-config-volume\") pod \"payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp\" (UID: \"8ee6272d-838e-4ff6-9e03-7ce29638a088\") " pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"
Sep 09 16:31:39.226164 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:39.226133    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpmxg\" (UniqueName: \"kubernetes.io/projected/8ee6272d-838e-4ff6-9e03-7ce29638a088-kube-api-access-cpmxg\") pod \"payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp\" (UID: \"8ee6272d-838e-4ff6-9e03-7ce29638a088\") " pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"
Sep 09 16:31:39.302623 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:39.302556    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"
Sep 09 16:31:39.571852 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:39.571829    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"]
Sep 09 16:31:39.574272 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:31:39.574242    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee6272d_838e_4ff6_9e03_7ce29638a088.slice/crio-43f1bbf538846948f491b2b288089ad521d464146f51e014d7448f6668c6e09e WatchSource:0}: Error finding container 43f1bbf538846948f491b2b288089ad521d464146f51e014d7448f6668c6e09e: Status 404 returned error can't find the container with id 43f1bbf538846948f491b2b288089ad521d464146f51e014d7448f6668c6e09e
Sep 09 16:31:39.853820 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:39.853729    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp" event={"ID":"8ee6272d-838e-4ff6-9e03-7ce29638a088","Type":"ContainerStarted","Data":"b120d0fd6730f546fcc219e7e42bfa5409c640acde98f56b5e2eca8be97cf418"}
Sep 09 16:31:39.853820 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:39.853785    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp" event={"ID":"8ee6272d-838e-4ff6-9e03-7ce29638a088","Type":"ContainerStarted","Data":"43f1bbf538846948f491b2b288089ad521d464146f51e014d7448f6668c6e09e"}
Sep 09 16:31:39.854033 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:39.853951    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"
Sep 09 16:31:39.871388 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:39.871347    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp" podStartSLOduration=1.871333016 podStartE2EDuration="1.871333016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:31:39.869831877 +0000 UTC m=+1375.805450307" watchObservedRunningTime="2026-09-09 16:31:39.871333016 +0000 UTC m=+1375.806951431"
Sep 09 16:31:44.646162 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:44.646132    2576 scope.go:117] "RemoveContainer" containerID="ef57c15f538e834a50d47729a1d582eb0352ed1f747688a8309aeefbed9f35d1"
Sep 09 16:31:44.654159 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:44.654140    2576 scope.go:117] "RemoveContainer" containerID="ecfd67783f29696d5434170f29408298db444e24b0035948e5e392fc6a972dee"
Sep 09 16:31:50.860850 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:31:50.860821    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"
Sep 09 16:32:07.913299 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:07.913269    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"]
Sep 09 16:32:07.917776 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:07.917759    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:07.920962 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:07.920941    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a0d4fa04\"/\"e2e-lifecycle-model-a0d4fa04-kserve-self-signed-certs\""
Sep 09 16:32:07.921146 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:07.921109    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a0d4fa04\"/\"kube-root-ca.crt\""
Sep 09 16:32:07.921146 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:07.921137    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a0d4fa04\"/\"openshift-service-ca.crt\""
Sep 09 16:32:07.921332 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:07.921115    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a0d4fa04\"/\"default-dockercfg-rqq7k\""
Sep 09 16:32:07.923881 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:07.923863    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"]
Sep 09 16:32:08.063789 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.063751    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-tmp-dir\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.063789 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.063795    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-dshm\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.064001 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.063837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-model-cache\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.064001 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.063858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml6br\" (UniqueName: \"kubernetes.io/projected/220897a0-f148-4cbb-8f02-7750d2edaafa-kube-api-access-ml6br\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.064001 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.063897    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-home\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.064001 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.063965    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/220897a0-f148-4cbb-8f02-7750d2edaafa-tls-certs\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.164996 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.164904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-tmp-dir\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.164996 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.164954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-dshm\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.165194 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.165005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-model-cache\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.165194 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.165038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml6br\" (UniqueName: \"kubernetes.io/projected/220897a0-f148-4cbb-8f02-7750d2edaafa-kube-api-access-ml6br\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.165194 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.165086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-home\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.165194 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.165125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/220897a0-f148-4cbb-8f02-7750d2edaafa-tls-certs\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.165392 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.165374    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-tmp-dir\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.165456 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.165421    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-model-cache\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.165512 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.165478    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-home\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.167402 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.167376    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-dshm\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.167648 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.167631    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/220897a0-f148-4cbb-8f02-7750d2edaafa-tls-certs\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.173237 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.173218    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml6br\" (UniqueName: \"kubernetes.io/projected/220897a0-f148-4cbb-8f02-7750d2edaafa-kube-api-access-ml6br\") pod \"e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") " pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.228436 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.228394    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:08.498710 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.498670    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"]
Sep 09 16:32:08.500913 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:32:08.500882    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod220897a0_f148_4cbb_8f02_7750d2edaafa.slice/crio-5c48b16956609e4ba9697d696d54a6d782d24c865dd6f23a2890979c627d1f11 WatchSource:0}: Error finding container 5c48b16956609e4ba9697d696d54a6d782d24c865dd6f23a2890979c627d1f11: Status 404 returned error can't find the container with id 5c48b16956609e4ba9697d696d54a6d782d24c865dd6f23a2890979c627d1f11
Sep 09 16:32:08.993243 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.993212    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm" event={"ID":"220897a0-f148-4cbb-8f02-7750d2edaafa","Type":"ContainerStarted","Data":"619d17f9586e3aa229d957b7befe554f2abdebc9ed1adf5a2324770d88b66a4b"}
Sep 09 16:32:08.993605 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:08.993255    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm" event={"ID":"220897a0-f148-4cbb-8f02-7750d2edaafa","Type":"ContainerStarted","Data":"5c48b16956609e4ba9697d696d54a6d782d24c865dd6f23a2890979c627d1f11"}
Sep 09 16:32:09.011731 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:09.011664    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm" podStartSLOduration=2.011650296 podStartE2EDuration="2.011650296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:32:09.009503233 +0000 UTC m=+1404.945121650" watchObservedRunningTime="2026-09-09 16:32:09.011650296 +0000 UTC m=+1404.947268758"
Sep 09 16:32:18.228744 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:18.228679    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:18.228744 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:18.228764    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:18.241595 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:18.241574    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:19.060255 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:19.060221    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:32:42.033789 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:42.033692    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64"]
Sep 09 16:32:42.037043 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:42.037026    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64"
Sep 09 16:32:42.039360 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:42.039340    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:32:42.039471 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:42.039347    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxghs\""
Sep 09 16:32:42.043812 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:42.043788    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64"]
Sep 09 16:32:42.162677 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:42.162641    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3325d886-e107-4bae-b5bd-38dc2fe00b1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64\" (UID: \"3325d886-e107-4bae-b5bd-38dc2fe00b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64"
Sep 09 16:32:42.263856 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:42.263822    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3325d886-e107-4bae-b5bd-38dc2fe00b1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64\" (UID: \"3325d886-e107-4bae-b5bd-38dc2fe00b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64"
Sep 09 16:32:42.264426 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:42.264406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3325d886-e107-4bae-b5bd-38dc2fe00b1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64\" (UID: \"3325d886-e107-4bae-b5bd-38dc2fe00b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64"
Sep 09 16:32:42.348977 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:42.348902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64"
Sep 09 16:32:42.642516 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:42.642453    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64"]
Sep 09 16:32:42.645271 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:32:42.645235    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3325d886_e107_4bae_b5bd_38dc2fe00b1d.slice/crio-6150a88abaf357e4e21263003f8f685d9f79809cb9d8b1b461a6c89213b314a5 WatchSource:0}: Error finding container 6150a88abaf357e4e21263003f8f685d9f79809cb9d8b1b461a6c89213b314a5: Status 404 returned error can't find the container with id 6150a88abaf357e4e21263003f8f685d9f79809cb9d8b1b461a6c89213b314a5
Sep 09 16:32:43.160131 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:43.160096    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64" event={"ID":"3325d886-e107-4bae-b5bd-38dc2fe00b1d","Type":"ContainerStarted","Data":"dc12d03e3cd8b8683d5cc924a15f2dd89d5c598b43da1010b45a15add2a35685"}
Sep 09 16:32:43.160500 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:43.160148    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64" event={"ID":"3325d886-e107-4bae-b5bd-38dc2fe00b1d","Type":"ContainerStarted","Data":"6150a88abaf357e4e21263003f8f685d9f79809cb9d8b1b461a6c89213b314a5"}
Sep 09 16:32:43.175203 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:43.175162    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64" podStartSLOduration=1.175147782 podStartE2EDuration="1.175147782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:32:43.173573996 +0000 UTC m=+1439.109192439" watchObservedRunningTime="2026-09-09 16:32:43.175147782 +0000 UTC m=+1439.110766242"
Sep 09 16:32:44.165841 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:44.165807    2576 generic.go:356] "Generic (PLEG): container finished" podID="3325d886-e107-4bae-b5bd-38dc2fe00b1d" containerID="dc12d03e3cd8b8683d5cc924a15f2dd89d5c598b43da1010b45a15add2a35685" exitCode=0
Sep 09 16:32:44.166216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:44.165895    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64" event={"ID":"3325d886-e107-4bae-b5bd-38dc2fe00b1d","Type":"ContainerDied","Data":"dc12d03e3cd8b8683d5cc924a15f2dd89d5c598b43da1010b45a15add2a35685"}
Sep 09 16:32:45.300408 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:45.300386    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64"
Sep 09 16:32:45.389953 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:45.389909    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3325d886-e107-4bae-b5bd-38dc2fe00b1d-maas-api-service-ca\") pod \"3325d886-e107-4bae-b5bd-38dc2fe00b1d\" (UID: \"3325d886-e107-4bae-b5bd-38dc2fe00b1d\") "
Sep 09 16:32:45.390248 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:45.390226    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3325d886-e107-4bae-b5bd-38dc2fe00b1d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3325d886-e107-4bae-b5bd-38dc2fe00b1d" (UID: "3325d886-e107-4bae-b5bd-38dc2fe00b1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:32:45.490791 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:45.490760    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3325d886-e107-4bae-b5bd-38dc2fe00b1d-maas-api-service-ca\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:32:46.176868 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:46.176832    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64"
Sep 09 16:32:46.177062 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:46.176867    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64" event={"ID":"3325d886-e107-4bae-b5bd-38dc2fe00b1d","Type":"ContainerDied","Data":"6150a88abaf357e4e21263003f8f685d9f79809cb9d8b1b461a6c89213b314a5"}
Sep 09 16:32:46.177062 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:46.176915    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6150a88abaf357e4e21263003f8f685d9f79809cb9d8b1b461a6c89213b314a5"
Sep 09 16:32:57.556047 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:57.556008    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"]
Sep 09 16:32:57.556519 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:57.556267    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp" podUID="8ee6272d-838e-4ff6-9e03-7ce29638a088" containerName="payload-processing" containerID="cri-o://b120d0fd6730f546fcc219e7e42bfa5409c640acde98f56b5e2eca8be97cf418" gracePeriod=30
Sep 09 16:32:57.723012 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:57.722982    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64"]
Sep 09 16:32:57.726061 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:57.726032    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0d4fa04-z5h64"]
Sep 09 16:32:57.799306 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:57.799282    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"
Sep 09 16:32:57.898385 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:57.898296    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee6272d-838e-4ff6-9e03-7ce29638a088-plugins-config-volume\") pod \"8ee6272d-838e-4ff6-9e03-7ce29638a088\" (UID: \"8ee6272d-838e-4ff6-9e03-7ce29638a088\") "
Sep 09 16:32:57.898385 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:57.898365    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpmxg\" (UniqueName: \"kubernetes.io/projected/8ee6272d-838e-4ff6-9e03-7ce29638a088-kube-api-access-cpmxg\") pod \"8ee6272d-838e-4ff6-9e03-7ce29638a088\" (UID: \"8ee6272d-838e-4ff6-9e03-7ce29638a088\") "
Sep 09 16:32:57.898681 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:57.898659    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee6272d-838e-4ff6-9e03-7ce29638a088-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ee6272d-838e-4ff6-9e03-7ce29638a088" (UID: "8ee6272d-838e-4ff6-9e03-7ce29638a088"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:32:57.900717 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:57.900677    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee6272d-838e-4ff6-9e03-7ce29638a088-kube-api-access-cpmxg" (OuterVolumeSpecName: "kube-api-access-cpmxg") pod "8ee6272d-838e-4ff6-9e03-7ce29638a088" (UID: "8ee6272d-838e-4ff6-9e03-7ce29638a088"). InnerVolumeSpecName "kube-api-access-cpmxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:32:57.999453 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:57.999413    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpmxg\" (UniqueName: \"kubernetes.io/projected/8ee6272d-838e-4ff6-9e03-7ce29638a088-kube-api-access-cpmxg\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:32:57.999453 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:57.999447    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee6272d-838e-4ff6-9e03-7ce29638a088-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:32:58.238014 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:58.237976    2576 generic.go:356] "Generic (PLEG): container finished" podID="8ee6272d-838e-4ff6-9e03-7ce29638a088" containerID="b120d0fd6730f546fcc219e7e42bfa5409c640acde98f56b5e2eca8be97cf418" exitCode=0
Sep 09 16:32:58.238205 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:58.238018    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"
Sep 09 16:32:58.238205 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:58.238060    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp" event={"ID":"8ee6272d-838e-4ff6-9e03-7ce29638a088","Type":"ContainerDied","Data":"b120d0fd6730f546fcc219e7e42bfa5409c640acde98f56b5e2eca8be97cf418"}
Sep 09 16:32:58.238205 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:58.238111    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp" event={"ID":"8ee6272d-838e-4ff6-9e03-7ce29638a088","Type":"ContainerDied","Data":"43f1bbf538846948f491b2b288089ad521d464146f51e014d7448f6668c6e09e"}
Sep 09 16:32:58.238205 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:58.238134    2576 scope.go:117] "RemoveContainer" containerID="b120d0fd6730f546fcc219e7e42bfa5409c640acde98f56b5e2eca8be97cf418"
Sep 09 16:32:58.247502 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:58.247485    2576 scope.go:117] "RemoveContainer" containerID="b120d0fd6730f546fcc219e7e42bfa5409c640acde98f56b5e2eca8be97cf418"
Sep 09 16:32:58.247797 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:32:58.247777    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b120d0fd6730f546fcc219e7e42bfa5409c640acde98f56b5e2eca8be97cf418\": container with ID starting with b120d0fd6730f546fcc219e7e42bfa5409c640acde98f56b5e2eca8be97cf418 not found: ID does not exist" containerID="b120d0fd6730f546fcc219e7e42bfa5409c640acde98f56b5e2eca8be97cf418"
Sep 09 16:32:58.247866 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:58.247803    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b120d0fd6730f546fcc219e7e42bfa5409c640acde98f56b5e2eca8be97cf418"} err="failed to get container status \"b120d0fd6730f546fcc219e7e42bfa5409c640acde98f56b5e2eca8be97cf418\": rpc error: code = NotFound desc = could not find container \"b120d0fd6730f546fcc219e7e42bfa5409c640acde98f56b5e2eca8be97cf418\": container with ID starting with b120d0fd6730f546fcc219e7e42bfa5409c640acde98f56b5e2eca8be97cf418 not found: ID does not exist"
Sep 09 16:32:58.258457 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:58.258432    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"]
Sep 09 16:32:58.262586 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:58.262564    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a0d4fa04-5b7b59c469-svnlp"]
Sep 09 16:32:58.553150 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:58.553070    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3325d886-e107-4bae-b5bd-38dc2fe00b1d" path="/var/lib/kubelet/pods/3325d886-e107-4bae-b5bd-38dc2fe00b1d/volumes"
Sep 09 16:32:58.553435 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:32:58.553422    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ee6272d-838e-4ff6-9e03-7ce29638a088" path="/var/lib/kubelet/pods/8ee6272d-838e-4ff6-9e03-7ce29638a088/volumes"
Sep 09 16:33:06.634286 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:06.634249    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"]
Sep 09 16:33:21.654110 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:21.654053    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm" podUID="220897a0-f148-4cbb-8f02-7750d2edaafa" containerName="main" containerID="cri-o://619d17f9586e3aa229d957b7befe554f2abdebc9ed1adf5a2324770d88b66a4b" gracePeriod=45
Sep 09 16:33:21.905516 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:21.905460    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:33:22.007989 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.007954    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/220897a0-f148-4cbb-8f02-7750d2edaafa-tls-certs\") pod \"220897a0-f148-4cbb-8f02-7750d2edaafa\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") "
Sep 09 16:33:22.008185 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.007998    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-tmp-dir\") pod \"220897a0-f148-4cbb-8f02-7750d2edaafa\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") "
Sep 09 16:33:22.008185 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.008023    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml6br\" (UniqueName: \"kubernetes.io/projected/220897a0-f148-4cbb-8f02-7750d2edaafa-kube-api-access-ml6br\") pod \"220897a0-f148-4cbb-8f02-7750d2edaafa\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") "
Sep 09 16:33:22.008185 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.008074    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-home\") pod \"220897a0-f148-4cbb-8f02-7750d2edaafa\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") "
Sep 09 16:33:22.008185 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.008102    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-model-cache\") pod \"220897a0-f148-4cbb-8f02-7750d2edaafa\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") "
Sep 09 16:33:22.008185 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.008148    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-dshm\") pod \"220897a0-f148-4cbb-8f02-7750d2edaafa\" (UID: \"220897a0-f148-4cbb-8f02-7750d2edaafa\") "
Sep 09 16:33:22.008457 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.008291    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "220897a0-f148-4cbb-8f02-7750d2edaafa" (UID: "220897a0-f148-4cbb-8f02-7750d2edaafa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:22.008457 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.008404    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-model-cache" (OuterVolumeSpecName: "model-cache") pod "220897a0-f148-4cbb-8f02-7750d2edaafa" (UID: "220897a0-f148-4cbb-8f02-7750d2edaafa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:22.008457 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.008417    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-home" (OuterVolumeSpecName: "home") pod "220897a0-f148-4cbb-8f02-7750d2edaafa" (UID: "220897a0-f148-4cbb-8f02-7750d2edaafa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:22.008457 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.008423    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:22.010340 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.010308    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-dshm" (OuterVolumeSpecName: "dshm") pod "220897a0-f148-4cbb-8f02-7750d2edaafa" (UID: "220897a0-f148-4cbb-8f02-7750d2edaafa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:22.010452 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.010391    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220897a0-f148-4cbb-8f02-7750d2edaafa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "220897a0-f148-4cbb-8f02-7750d2edaafa" (UID: "220897a0-f148-4cbb-8f02-7750d2edaafa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:33:22.010493 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.010445    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220897a0-f148-4cbb-8f02-7750d2edaafa-kube-api-access-ml6br" (OuterVolumeSpecName: "kube-api-access-ml6br") pod "220897a0-f148-4cbb-8f02-7750d2edaafa" (UID: "220897a0-f148-4cbb-8f02-7750d2edaafa"). InnerVolumeSpecName "kube-api-access-ml6br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:33:22.108944 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.108917    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/220897a0-f148-4cbb-8f02-7750d2edaafa-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:22.108944 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.108941    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml6br\" (UniqueName: \"kubernetes.io/projected/220897a0-f148-4cbb-8f02-7750d2edaafa-kube-api-access-ml6br\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:22.109125 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.108951    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:22.109125 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.108960    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:22.109125 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.108969    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/220897a0-f148-4cbb-8f02-7750d2edaafa-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:22.349440 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.349404    2576 generic.go:356] "Generic (PLEG): container finished" podID="220897a0-f148-4cbb-8f02-7750d2edaafa" containerID="619d17f9586e3aa229d957b7befe554f2abdebc9ed1adf5a2324770d88b66a4b" exitCode=0
Sep 09 16:33:22.349628 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.349454    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"
Sep 09 16:33:22.349628 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.349472    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm" event={"ID":"220897a0-f148-4cbb-8f02-7750d2edaafa","Type":"ContainerDied","Data":"619d17f9586e3aa229d957b7befe554f2abdebc9ed1adf5a2324770d88b66a4b"}
Sep 09 16:33:22.349628 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.349511    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm" event={"ID":"220897a0-f148-4cbb-8f02-7750d2edaafa","Type":"ContainerDied","Data":"5c48b16956609e4ba9697d696d54a6d782d24c865dd6f23a2890979c627d1f11"}
Sep 09 16:33:22.349628 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.349532    2576 scope.go:117] "RemoveContainer" containerID="619d17f9586e3aa229d957b7befe554f2abdebc9ed1adf5a2324770d88b66a4b"
Sep 09 16:33:22.358243 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.358225    2576 scope.go:117] "RemoveContainer" containerID="619d17f9586e3aa229d957b7befe554f2abdebc9ed1adf5a2324770d88b66a4b"
Sep 09 16:33:22.358492 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:33:22.358475    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619d17f9586e3aa229d957b7befe554f2abdebc9ed1adf5a2324770d88b66a4b\": container with ID starting with 619d17f9586e3aa229d957b7befe554f2abdebc9ed1adf5a2324770d88b66a4b not found: ID does not exist" containerID="619d17f9586e3aa229d957b7befe554f2abdebc9ed1adf5a2324770d88b66a4b"
Sep 09 16:33:22.358540 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.358501    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619d17f9586e3aa229d957b7befe554f2abdebc9ed1adf5a2324770d88b66a4b"} err="failed to get container status \"619d17f9586e3aa229d957b7befe554f2abdebc9ed1adf5a2324770d88b66a4b\": rpc error: code = NotFound desc = could not find container \"619d17f9586e3aa229d957b7befe554f2abdebc9ed1adf5a2324770d88b66a4b\": container with ID starting with 619d17f9586e3aa229d957b7befe554f2abdebc9ed1adf5a2324770d88b66a4b not found: ID does not exist"
Sep 09 16:33:22.370152 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.370125    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"]
Sep 09 16:33:22.373477 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.373459    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a0d4fa04/e2e-lifecycle-model-a0d4fa04-kserve-7b96795b7f-tvkpm"]
Sep 09 16:33:22.553534 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:22.553501    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="220897a0-f148-4cbb-8f02-7750d2edaafa" path="/var/lib/kubelet/pods/220897a0-f148-4cbb-8f02-7750d2edaafa/volumes"
Sep 09 16:33:36.209710 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:36.209663    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"]
Sep 09 16:33:37.198206 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:37.198164    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"]
Sep 09 16:33:40.194629 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.194595    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"]
Sep 09 16:33:40.195195 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.195082    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3325d886-e107-4bae-b5bd-38dc2fe00b1d" containerName="revoke-keys"
Sep 09 16:33:40.195195 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.195102    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3325d886-e107-4bae-b5bd-38dc2fe00b1d" containerName="revoke-keys"
Sep 09 16:33:40.195195 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.195121    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee6272d-838e-4ff6-9e03-7ce29638a088" containerName="payload-processing"
Sep 09 16:33:40.195195 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.195129    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee6272d-838e-4ff6-9e03-7ce29638a088" containerName="payload-processing"
Sep 09 16:33:40.195195 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.195143    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="220897a0-f148-4cbb-8f02-7750d2edaafa" containerName="main"
Sep 09 16:33:40.195195 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.195151    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="220897a0-f148-4cbb-8f02-7750d2edaafa" containerName="main"
Sep 09 16:33:40.195500 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.195251    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="220897a0-f148-4cbb-8f02-7750d2edaafa" containerName="main"
Sep 09 16:33:40.195500 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.195266    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ee6272d-838e-4ff6-9e03-7ce29638a088" containerName="payload-processing"
Sep 09 16:33:40.195500 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.195277    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3325d886-e107-4bae-b5bd-38dc2fe00b1d" containerName="revoke-keys"
Sep 09 16:33:40.199585 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.199565    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.202057 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.202032    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-041bf1\"/\"sub-test-model-041bf1-kserve-self-signed-certs\""
Sep 09 16:33:40.209374 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.209354    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"]
Sep 09 16:33:40.268656 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.268612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94bfeea1-439f-4d47-b63e-0e7ba87bced1-tls-certs\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.268876 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.268716    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-tmp-dir\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.268876 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.268799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-home\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.268876 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.268831    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-model-cache\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.268876 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.268849    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmst\" (UniqueName: \"kubernetes.io/projected/94bfeea1-439f-4d47-b63e-0e7ba87bced1-kube-api-access-tbmst\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.269114 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.268911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-dshm\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.369825 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.369784    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-home\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.369825 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.369824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-model-cache\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.370037 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.369840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmst\" (UniqueName: \"kubernetes.io/projected/94bfeea1-439f-4d47-b63e-0e7ba87bced1-kube-api-access-tbmst\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.370037 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.369956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-dshm\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.370127 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.370047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94bfeea1-439f-4d47-b63e-0e7ba87bced1-tls-certs\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.370127 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.370116    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-tmp-dir\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.370242 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.370220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-model-cache\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.370296 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.370274    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-home\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.370422 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.370405    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-tmp-dir\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.372270 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.372250    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-dshm\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.372664 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.372640    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94bfeea1-439f-4d47-b63e-0e7ba87bced1-tls-certs\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.377494 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.377472    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmst\" (UniqueName: \"kubernetes.io/projected/94bfeea1-439f-4d47-b63e-0e7ba87bced1-kube-api-access-tbmst\") pod \"sub-test-model-041bf1-kserve-8645cbb6c5-m5998\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.510357 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.510323    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:40.779665 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:40.779643    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"]
Sep 09 16:33:40.782351 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:33:40.782321    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94bfeea1_439f_4d47_b63e_0e7ba87bced1.slice/crio-1f7c29cee8f76e16efafae1f0a1dbbba3ef349a9c04910685b87274c5bdb21cd WatchSource:0}: Error finding container 1f7c29cee8f76e16efafae1f0a1dbbba3ef349a9c04910685b87274c5bdb21cd: Status 404 returned error can't find the container with id 1f7c29cee8f76e16efafae1f0a1dbbba3ef349a9c04910685b87274c5bdb21cd
Sep 09 16:33:41.438504 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:41.438467    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998" event={"ID":"94bfeea1-439f-4d47-b63e-0e7ba87bced1","Type":"ContainerStarted","Data":"4735bcfc7eaad35566b1dac6da55a9bfe87770097f0185adde1611b4ddb77da4"}
Sep 09 16:33:41.438504 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:41.438508    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998" event={"ID":"94bfeea1-439f-4d47-b63e-0e7ba87bced1","Type":"ContainerStarted","Data":"1f7c29cee8f76e16efafae1f0a1dbbba3ef349a9c04910685b87274c5bdb21cd"}
Sep 09 16:33:41.455467 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:41.455417    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998" podStartSLOduration=1.455402276 podStartE2EDuration="1.455402276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:33:41.455063618 +0000 UTC m=+1497.390682046" watchObservedRunningTime="2026-09-09 16:33:41.455402276 +0000 UTC m=+1497.391020692"
Sep 09 16:33:50.510931 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:50.510899    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:50.511326 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:50.510947    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:50.523667 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:50.523643    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:51.229934 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.229875    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8" podUID="feb41bc3-529d-45c9-ad64-83a36dea7500" containerName="main" containerID="cri-o://6668012ee5c6724f3e94bfa9090a3cee85f6a62037a3de0d360f6e3986b02d80" gracePeriod=45
Sep 09 16:33:51.483079 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.483020    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:33:51.484190 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.484171    2576 generic.go:356] "Generic (PLEG): container finished" podID="feb41bc3-529d-45c9-ad64-83a36dea7500" containerID="6668012ee5c6724f3e94bfa9090a3cee85f6a62037a3de0d360f6e3986b02d80" exitCode=0
Sep 09 16:33:51.484306 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.484239    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8" event={"ID":"feb41bc3-529d-45c9-ad64-83a36dea7500","Type":"ContainerDied","Data":"6668012ee5c6724f3e94bfa9090a3cee85f6a62037a3de0d360f6e3986b02d80"}
Sep 09 16:33:51.484306 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.484277    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8" event={"ID":"feb41bc3-529d-45c9-ad64-83a36dea7500","Type":"ContainerDied","Data":"cddaa6bc1a9da60bdef915311bf590cbdcd9eb82acd206de0f1f9778fde153c9"}
Sep 09 16:33:51.484306 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.484294    2576 scope.go:117] "RemoveContainer" containerID="6668012ee5c6724f3e94bfa9090a3cee85f6a62037a3de0d360f6e3986b02d80"
Sep 09 16:33:51.493340 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.493319    2576 scope.go:117] "RemoveContainer" containerID="6668012ee5c6724f3e94bfa9090a3cee85f6a62037a3de0d360f6e3986b02d80"
Sep 09 16:33:51.493641 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:33:51.493617    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6668012ee5c6724f3e94bfa9090a3cee85f6a62037a3de0d360f6e3986b02d80\": container with ID starting with 6668012ee5c6724f3e94bfa9090a3cee85f6a62037a3de0d360f6e3986b02d80 not found: ID does not exist" containerID="6668012ee5c6724f3e94bfa9090a3cee85f6a62037a3de0d360f6e3986b02d80"
Sep 09 16:33:51.493767 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.493652    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6668012ee5c6724f3e94bfa9090a3cee85f6a62037a3de0d360f6e3986b02d80"} err="failed to get container status \"6668012ee5c6724f3e94bfa9090a3cee85f6a62037a3de0d360f6e3986b02d80\": rpc error: code = NotFound desc = could not find container \"6668012ee5c6724f3e94bfa9090a3cee85f6a62037a3de0d360f6e3986b02d80\": container with ID starting with 6668012ee5c6724f3e94bfa9090a3cee85f6a62037a3de0d360f6e3986b02d80 not found: ID does not exist"
Sep 09 16:33:51.495814 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.495786    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:33:51.569437 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.569405    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-dshm\") pod \"feb41bc3-529d-45c9-ad64-83a36dea7500\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") "
Sep 09 16:33:51.569921 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.569451    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb41bc3-529d-45c9-ad64-83a36dea7500-tls-certs\") pod \"feb41bc3-529d-45c9-ad64-83a36dea7500\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") "
Sep 09 16:33:51.569921 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.569484    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-home\") pod \"feb41bc3-529d-45c9-ad64-83a36dea7500\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") "
Sep 09 16:33:51.569921 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.569509    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-tmp-dir\") pod \"feb41bc3-529d-45c9-ad64-83a36dea7500\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") "
Sep 09 16:33:51.569921 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.569537    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-model-cache\") pod \"feb41bc3-529d-45c9-ad64-83a36dea7500\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") "
Sep 09 16:33:51.569921 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.569572    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dswh\" (UniqueName: \"kubernetes.io/projected/feb41bc3-529d-45c9-ad64-83a36dea7500-kube-api-access-5dswh\") pod \"feb41bc3-529d-45c9-ad64-83a36dea7500\" (UID: \"feb41bc3-529d-45c9-ad64-83a36dea7500\") "
Sep 09 16:33:51.569921 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.569851    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-home" (OuterVolumeSpecName: "home") pod "feb41bc3-529d-45c9-ad64-83a36dea7500" (UID: "feb41bc3-529d-45c9-ad64-83a36dea7500"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:51.569921 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.569865    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "feb41bc3-529d-45c9-ad64-83a36dea7500" (UID: "feb41bc3-529d-45c9-ad64-83a36dea7500"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:51.570456 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.569944    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:51.570456 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.569966    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:51.570456 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.570188    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-model-cache" (OuterVolumeSpecName: "model-cache") pod "feb41bc3-529d-45c9-ad64-83a36dea7500" (UID: "feb41bc3-529d-45c9-ad64-83a36dea7500"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:51.571943 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.571921    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-dshm" (OuterVolumeSpecName: "dshm") pod "feb41bc3-529d-45c9-ad64-83a36dea7500" (UID: "feb41bc3-529d-45c9-ad64-83a36dea7500"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:51.572341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.572319    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb41bc3-529d-45c9-ad64-83a36dea7500-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "feb41bc3-529d-45c9-ad64-83a36dea7500" (UID: "feb41bc3-529d-45c9-ad64-83a36dea7500"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:33:51.572389 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.572342    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb41bc3-529d-45c9-ad64-83a36dea7500-kube-api-access-5dswh" (OuterVolumeSpecName: "kube-api-access-5dswh") pod "feb41bc3-529d-45c9-ad64-83a36dea7500" (UID: "feb41bc3-529d-45c9-ad64-83a36dea7500"). InnerVolumeSpecName "kube-api-access-5dswh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:33:51.671413 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.671349    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:51.671413 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.671386    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dswh\" (UniqueName: \"kubernetes.io/projected/feb41bc3-529d-45c9-ad64-83a36dea7500-kube-api-access-5dswh\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:51.671413 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.671401    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb41bc3-529d-45c9-ad64-83a36dea7500-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:51.671413 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:51.671414    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb41bc3-529d-45c9-ad64-83a36dea7500-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:52.218328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.218289    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl" podUID="eeffab74-534d-4cb4-9af1-40a5d38eb536" containerName="main" containerID="cri-o://209fc2d5b004581b91fb2725df9245f7f434a885f5984038df6c6fe4e75f2a1e" gracePeriod=45
Sep 09 16:33:52.479809 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.479782    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:33:52.490356 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.490331    2576 generic.go:356] "Generic (PLEG): container finished" podID="eeffab74-534d-4cb4-9af1-40a5d38eb536" containerID="209fc2d5b004581b91fb2725df9245f7f434a885f5984038df6c6fe4e75f2a1e" exitCode=0
Sep 09 16:33:52.490489 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.490376    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"
Sep 09 16:33:52.490489 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.490415    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl" event={"ID":"eeffab74-534d-4cb4-9af1-40a5d38eb536","Type":"ContainerDied","Data":"209fc2d5b004581b91fb2725df9245f7f434a885f5984038df6c6fe4e75f2a1e"}
Sep 09 16:33:52.490489 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.490461    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl" event={"ID":"eeffab74-534d-4cb4-9af1-40a5d38eb536","Type":"ContainerDied","Data":"947c22ece67c8ec9e38159446c34d9aeacf416f169b78e538e4b9a23e0370ed3"}
Sep 09 16:33:52.490489 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.490485    2576 scope.go:117] "RemoveContainer" containerID="209fc2d5b004581b91fb2725df9245f7f434a885f5984038df6c6fe4e75f2a1e"
Sep 09 16:33:52.494037 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.494016    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"
Sep 09 16:33:52.504109 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.504090    2576 scope.go:117] "RemoveContainer" containerID="209fc2d5b004581b91fb2725df9245f7f434a885f5984038df6c6fe4e75f2a1e"
Sep 09 16:33:52.504393 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:33:52.504374    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209fc2d5b004581b91fb2725df9245f7f434a885f5984038df6c6fe4e75f2a1e\": container with ID starting with 209fc2d5b004581b91fb2725df9245f7f434a885f5984038df6c6fe4e75f2a1e not found: ID does not exist" containerID="209fc2d5b004581b91fb2725df9245f7f434a885f5984038df6c6fe4e75f2a1e"
Sep 09 16:33:52.504478 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.504403    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209fc2d5b004581b91fb2725df9245f7f434a885f5984038df6c6fe4e75f2a1e"} err="failed to get container status \"209fc2d5b004581b91fb2725df9245f7f434a885f5984038df6c6fe4e75f2a1e\": rpc error: code = NotFound desc = could not find container \"209fc2d5b004581b91fb2725df9245f7f434a885f5984038df6c6fe4e75f2a1e\": container with ID starting with 209fc2d5b004581b91fb2725df9245f7f434a885f5984038df6c6fe4e75f2a1e not found: ID does not exist"
Sep 09 16:33:52.515616 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.515585    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"]
Sep 09 16:33:52.519553 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.519533    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/auth-test-model-041bf1-kserve-8c48fc67f-pzvl8"]
Sep 09 16:33:52.553640 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.553606    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feb41bc3-529d-45c9-ad64-83a36dea7500" path="/var/lib/kubelet/pods/feb41bc3-529d-45c9-ad64-83a36dea7500/volumes"
Sep 09 16:33:52.579569 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.579545    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-tmp-dir\") pod \"eeffab74-534d-4cb4-9af1-40a5d38eb536\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") "
Sep 09 16:33:52.579910 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.579578    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-home\") pod \"eeffab74-534d-4cb4-9af1-40a5d38eb536\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") "
Sep 09 16:33:52.579910 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.579647    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-dshm\") pod \"eeffab74-534d-4cb4-9af1-40a5d38eb536\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") "
Sep 09 16:33:52.579910 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.579674    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-model-cache\") pod \"eeffab74-534d-4cb4-9af1-40a5d38eb536\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") "
Sep 09 16:33:52.579910 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.579755    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeffab74-534d-4cb4-9af1-40a5d38eb536-tls-certs\") pod \"eeffab74-534d-4cb4-9af1-40a5d38eb536\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") "
Sep 09 16:33:52.579910 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.579782    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb2gp\" (UniqueName: \"kubernetes.io/projected/eeffab74-534d-4cb4-9af1-40a5d38eb536-kube-api-access-zb2gp\") pod \"eeffab74-534d-4cb4-9af1-40a5d38eb536\" (UID: \"eeffab74-534d-4cb4-9af1-40a5d38eb536\") "
Sep 09 16:33:52.579910 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.579849    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eeffab74-534d-4cb4-9af1-40a5d38eb536" (UID: "eeffab74-534d-4cb4-9af1-40a5d38eb536"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:52.579910 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.579885    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-home" (OuterVolumeSpecName: "home") pod "eeffab74-534d-4cb4-9af1-40a5d38eb536" (UID: "eeffab74-534d-4cb4-9af1-40a5d38eb536"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:52.580250 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.580025    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-model-cache" (OuterVolumeSpecName: "model-cache") pod "eeffab74-534d-4cb4-9af1-40a5d38eb536" (UID: "eeffab74-534d-4cb4-9af1-40a5d38eb536"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:52.580250 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.580106    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:52.580250 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.580124    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:52.580250 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.580137    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:52.582055 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.582037    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-dshm" (OuterVolumeSpecName: "dshm") pod "eeffab74-534d-4cb4-9af1-40a5d38eb536" (UID: "eeffab74-534d-4cb4-9af1-40a5d38eb536"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:52.582055 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.582041    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeffab74-534d-4cb4-9af1-40a5d38eb536-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eeffab74-534d-4cb4-9af1-40a5d38eb536" (UID: "eeffab74-534d-4cb4-9af1-40a5d38eb536"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:33:52.582181 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.582143    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeffab74-534d-4cb4-9af1-40a5d38eb536-kube-api-access-zb2gp" (OuterVolumeSpecName: "kube-api-access-zb2gp") pod "eeffab74-534d-4cb4-9af1-40a5d38eb536" (UID: "eeffab74-534d-4cb4-9af1-40a5d38eb536"). InnerVolumeSpecName "kube-api-access-zb2gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:33:52.681298 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.681261    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeffab74-534d-4cb4-9af1-40a5d38eb536-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:52.681298 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.681291    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeffab74-534d-4cb4-9af1-40a5d38eb536-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:52.681298 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.681301    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb2gp\" (UniqueName: \"kubernetes.io/projected/eeffab74-534d-4cb4-9af1-40a5d38eb536-kube-api-access-zb2gp\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:33:52.810402 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.810376    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"]
Sep 09 16:33:52.813591 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:52.813565    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/auth-test-model-9520a4-kserve-7b4b94958c-kdpvl"]
Sep 09 16:33:54.558057 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:33:54.558019    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeffab74-534d-4cb4-9af1-40a5d38eb536" path="/var/lib/kubelet/pods/eeffab74-534d-4cb4-9af1-40a5d38eb536/volumes"
Sep 09 16:34:02.409798 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.409764    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"]
Sep 09 16:34:02.410171 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.410081    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb41bc3-529d-45c9-ad64-83a36dea7500" containerName="main"
Sep 09 16:34:02.410171 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.410091    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb41bc3-529d-45c9-ad64-83a36dea7500" containerName="main"
Sep 09 16:34:02.410171 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.410103    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeffab74-534d-4cb4-9af1-40a5d38eb536" containerName="main"
Sep 09 16:34:02.410171 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.410109    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeffab74-534d-4cb4-9af1-40a5d38eb536" containerName="main"
Sep 09 16:34:02.410171 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.410167    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeffab74-534d-4cb4-9af1-40a5d38eb536" containerName="main"
Sep 09 16:34:02.410344 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.410178    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="feb41bc3-529d-45c9-ad64-83a36dea7500" containerName="main"
Sep 09 16:34:02.414503 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.414487    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.416918 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.416897    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9520a4\"/\"kube-root-ca.crt\""
Sep 09 16:34:02.418271 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.418251    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9520a4\"/\"default-dockercfg-qlph4\""
Sep 09 16:34:02.418370 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.418291    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9520a4\"/\"sub-test-model-9520a4-kserve-self-signed-certs\""
Sep 09 16:34:02.418522 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.418504    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9520a4\"/\"openshift-service-ca.crt\""
Sep 09 16:34:02.423285 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.423264    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"]
Sep 09 16:34:02.470986 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.470952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-home\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.471155 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.471010    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da41616-bd38-4dac-8892-9b17f65d94cf-tls-certs\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.471155 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.471046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-model-cache\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.471155 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.471118    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-667gk\" (UniqueName: \"kubernetes.io/projected/9da41616-bd38-4dac-8892-9b17f65d94cf-kube-api-access-667gk\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.471352 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.471159    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-dshm\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.471352 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.471184    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-tmp-dir\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.571980 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.571948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-tmp-dir\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.572134 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.572007    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-home\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.572134 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.572032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da41616-bd38-4dac-8892-9b17f65d94cf-tls-certs\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.572134 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.572071    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-model-cache\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.572294 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.572138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667gk\" (UniqueName: \"kubernetes.io/projected/9da41616-bd38-4dac-8892-9b17f65d94cf-kube-api-access-667gk\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.572294 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.572183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-dshm\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.572436 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.572416    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-tmp-dir\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.572543 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.572432    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-home\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.572543 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.572497    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-model-cache\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.574573 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.574555    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-dshm\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.574760 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.574745    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da41616-bd38-4dac-8892-9b17f65d94cf-tls-certs\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.585154 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.585134    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-667gk\" (UniqueName: \"kubernetes.io/projected/9da41616-bd38-4dac-8892-9b17f65d94cf-kube-api-access-667gk\") pod \"sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.724301 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.724260    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:02.993761 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:02.993659    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"]
Sep 09 16:34:02.997177 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:34:02.997152    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9da41616_bd38_4dac_8892_9b17f65d94cf.slice/crio-8c82e36816788873a0442494a9be0ba3c94030d84313e812e2270a485e4fc559 WatchSource:0}: Error finding container 8c82e36816788873a0442494a9be0ba3c94030d84313e812e2270a485e4fc559: Status 404 returned error can't find the container with id 8c82e36816788873a0442494a9be0ba3c94030d84313e812e2270a485e4fc559
Sep 09 16:34:03.546952 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:03.546915    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg" event={"ID":"9da41616-bd38-4dac-8892-9b17f65d94cf","Type":"ContainerStarted","Data":"a33546564888041a79b51ce4e0a2c24532d66b2147f5940e72bb4f8938d3a4a3"}
Sep 09 16:34:03.546952 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:03.546957    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg" event={"ID":"9da41616-bd38-4dac-8892-9b17f65d94cf","Type":"ContainerStarted","Data":"8c82e36816788873a0442494a9be0ba3c94030d84313e812e2270a485e4fc559"}
Sep 09 16:34:03.564319 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:03.564269    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg" podStartSLOduration=1.5642535610000001 podStartE2EDuration="1.564253561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:34:03.562468581 +0000 UTC m=+1519.498087016" watchObservedRunningTime="2026-09-09 16:34:03.564253561 +0000 UTC m=+1519.499871976"
Sep 09 16:34:12.724675 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:12.724581    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:12.724675 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:12.724631    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:12.737567 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:12.737546    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:13.606087 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:13.606063    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:34:32.856213 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:32.856177    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"]
Sep 09 16:34:32.859549 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:32.859534    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"
Sep 09 16:34:32.862364 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:32.862342    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5d8327-dockercfg-cg9qx\""
Sep 09 16:34:32.862486 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:32.862420    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5d8327\""
Sep 09 16:34:32.877725 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:32.876098    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"]
Sep 09 16:34:32.915906 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:32.915875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373194db-188d-4dcc-90b9-10f7941069d9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf\" (UID: \"373194db-188d-4dcc-90b9-10f7941069d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"
Sep 09 16:34:32.916024 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:32.915934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxj55\" (UniqueName: \"kubernetes.io/projected/373194db-188d-4dcc-90b9-10f7941069d9-kube-api-access-qxj55\") pod \"payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf\" (UID: \"373194db-188d-4dcc-90b9-10f7941069d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"
Sep 09 16:34:33.017235 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:33.017182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373194db-188d-4dcc-90b9-10f7941069d9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf\" (UID: \"373194db-188d-4dcc-90b9-10f7941069d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"
Sep 09 16:34:33.017405 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:33.017289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxj55\" (UniqueName: \"kubernetes.io/projected/373194db-188d-4dcc-90b9-10f7941069d9-kube-api-access-qxj55\") pod \"payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf\" (UID: \"373194db-188d-4dcc-90b9-10f7941069d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"
Sep 09 16:34:33.017907 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:33.017886    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373194db-188d-4dcc-90b9-10f7941069d9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf\" (UID: \"373194db-188d-4dcc-90b9-10f7941069d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"
Sep 09 16:34:33.025221 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:33.025203    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxj55\" (UniqueName: \"kubernetes.io/projected/373194db-188d-4dcc-90b9-10f7941069d9-kube-api-access-qxj55\") pod \"payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf\" (UID: \"373194db-188d-4dcc-90b9-10f7941069d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"
Sep 09 16:34:33.169547 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:33.169461    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"
Sep 09 16:34:33.440026 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:33.440001    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"]
Sep 09 16:34:33.442106 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:34:33.442078    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373194db_188d_4dcc_90b9_10f7941069d9.slice/crio-effdcd987f917abd60b1b2351f3216ed68afcf865d68d529c081a95050f85357 WatchSource:0}: Error finding container effdcd987f917abd60b1b2351f3216ed68afcf865d68d529c081a95050f85357: Status 404 returned error can't find the container with id effdcd987f917abd60b1b2351f3216ed68afcf865d68d529c081a95050f85357
Sep 09 16:34:33.688035 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:33.687997    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf" event={"ID":"373194db-188d-4dcc-90b9-10f7941069d9","Type":"ContainerStarted","Data":"14a4d245c33f6e30379f5bfacbc0b89c3dd597f17f90bb19e52c516faaf39ba0"}
Sep 09 16:34:33.688213 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:33.688056    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf" event={"ID":"373194db-188d-4dcc-90b9-10f7941069d9","Type":"ContainerStarted","Data":"effdcd987f917abd60b1b2351f3216ed68afcf865d68d529c081a95050f85357"}
Sep 09 16:34:33.688213 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:33.688083    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"
Sep 09 16:34:33.705559 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:33.705467    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf" podStartSLOduration=1.705447767 podStartE2EDuration="1.705447767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:34:33.703784093 +0000 UTC m=+1549.639402532" watchObservedRunningTime="2026-09-09 16:34:33.705447767 +0000 UTC m=+1549.641066182"
Sep 09 16:34:44.696101 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:44.696056    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"
Sep 09 16:34:54.295572 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:54.295540    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"]
Sep 09 16:34:55.336280 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:34:55.336249    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"]
Sep 09 16:35:02.134906 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.134862    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"]
Sep 09 16:35:02.139617 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.139600    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"
Sep 09 16:35:02.142147 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.142125    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ab78ef\""
Sep 09 16:35:02.142262 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.142164    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ab78ef-dockercfg-qrtjb\""
Sep 09 16:35:02.149856 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.149834    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"]
Sep 09 16:35:02.165304 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.165274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985e28bd-b0f2-4f3a-a532-ea24bbb04e05-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw\" (UID: \"985e28bd-b0f2-4f3a-a532-ea24bbb04e05\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"
Sep 09 16:35:02.165437 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.165315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn62z\" (UniqueName: \"kubernetes.io/projected/985e28bd-b0f2-4f3a-a532-ea24bbb04e05-kube-api-access-dn62z\") pod \"payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw\" (UID: \"985e28bd-b0f2-4f3a-a532-ea24bbb04e05\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"
Sep 09 16:35:02.265960 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.265913    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985e28bd-b0f2-4f3a-a532-ea24bbb04e05-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw\" (UID: \"985e28bd-b0f2-4f3a-a532-ea24bbb04e05\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"
Sep 09 16:35:02.266120 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.265977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn62z\" (UniqueName: \"kubernetes.io/projected/985e28bd-b0f2-4f3a-a532-ea24bbb04e05-kube-api-access-dn62z\") pod \"payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw\" (UID: \"985e28bd-b0f2-4f3a-a532-ea24bbb04e05\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"
Sep 09 16:35:02.266602 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.266581    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985e28bd-b0f2-4f3a-a532-ea24bbb04e05-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw\" (UID: \"985e28bd-b0f2-4f3a-a532-ea24bbb04e05\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"
Sep 09 16:35:02.275373 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.275347    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn62z\" (UniqueName: \"kubernetes.io/projected/985e28bd-b0f2-4f3a-a532-ea24bbb04e05-kube-api-access-dn62z\") pod \"payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw\" (UID: \"985e28bd-b0f2-4f3a-a532-ea24bbb04e05\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"
Sep 09 16:35:02.451252 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.451162    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"
Sep 09 16:35:02.730246 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.730215    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"]
Sep 09 16:35:02.730418 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:35:02.730282    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985e28bd_b0f2_4f3a_a532_ea24bbb04e05.slice/crio-a9421a7759276a5dcb5dc8849fe840f523fea254f61a83780e728730a286c6e9 WatchSource:0}: Error finding container a9421a7759276a5dcb5dc8849fe840f523fea254f61a83780e728730a286c6e9: Status 404 returned error can't find the container with id a9421a7759276a5dcb5dc8849fe840f523fea254f61a83780e728730a286c6e9
Sep 09 16:35:02.822933 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.822900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw" event={"ID":"985e28bd-b0f2-4f3a-a532-ea24bbb04e05","Type":"ContainerStarted","Data":"778db26d7d3b9bc287b15584d0454e986fb412ece912406476aa867dfd7340e8"}
Sep 09 16:35:02.823104 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.822949    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw" event={"ID":"985e28bd-b0f2-4f3a-a532-ea24bbb04e05","Type":"ContainerStarted","Data":"a9421a7759276a5dcb5dc8849fe840f523fea254f61a83780e728730a286c6e9"}
Sep 09 16:35:02.823104 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.823045    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"
Sep 09 16:35:02.838613 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:02.838565    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw" podStartSLOduration=0.838546696 podStartE2EDuration="838.546696ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:35:02.837394284 +0000 UTC m=+1578.773012712" watchObservedRunningTime="2026-09-09 16:35:02.838546696 +0000 UTC m=+1578.774165114"
Sep 09 16:35:08.805061 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:08.805020    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"]
Sep 09 16:35:08.808669 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:08.808651    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:08.810984 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:08.810961    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-041bf1\"/\"rate-test-model-041bf1-kserve-self-signed-certs\""
Sep 09 16:35:08.818764 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:08.818741    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"]
Sep 09 16:35:08.923234 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:08.923199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-model-cache\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:08.923234 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:08.923229    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-tmp-dir\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:08.923451 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:08.923269    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wrg4\" (UniqueName: \"kubernetes.io/projected/bdbc0851-7ea9-41e3-b8db-c39c446300da-kube-api-access-5wrg4\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:08.923451 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:08.923334    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-dshm\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:08.923451 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:08.923412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdbc0851-7ea9-41e3-b8db-c39c446300da-tls-certs\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:08.923451 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:08.923440    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-home\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:09.024552 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.024512    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-dshm\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:09.024776 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.024609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdbc0851-7ea9-41e3-b8db-c39c446300da-tls-certs\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:09.024776 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.024645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-home\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:09.024776 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.024671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-model-cache\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:09.024776 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.024722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-tmp-dir\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:09.024999 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.024786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wrg4\" (UniqueName: \"kubernetes.io/projected/bdbc0851-7ea9-41e3-b8db-c39c446300da-kube-api-access-5wrg4\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:09.025151 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.025129    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-model-cache\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:09.025202 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.025158    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-tmp-dir\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:09.025202 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.025118    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-home\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:09.027550 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.027527    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-dshm\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:09.027801 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.027783    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdbc0851-7ea9-41e3-b8db-c39c446300da-tls-certs\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:09.033536 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.033515    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wrg4\" (UniqueName: \"kubernetes.io/projected/bdbc0851-7ea9-41e3-b8db-c39c446300da-kube-api-access-5wrg4\") pod \"rate-test-model-041bf1-kserve-7978954c79-vl54p\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") " pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:09.118752 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.118648    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:09.315360 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.315305    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998" podUID="94bfeea1-439f-4d47-b63e-0e7ba87bced1" containerName="main" containerID="cri-o://4735bcfc7eaad35566b1dac6da55a9bfe87770097f0185adde1611b4ddb77da4" gracePeriod=45
Sep 09 16:35:09.394092 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.393998    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"]
Sep 09 16:35:09.554292 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.554266    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:35:09.631634 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.631603    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94bfeea1-439f-4d47-b63e-0e7ba87bced1-tls-certs\") pod \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") "
Sep 09 16:35:09.631814 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.631646    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-home\") pod \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") "
Sep 09 16:35:09.631814 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.631724    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-model-cache\") pod \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") "
Sep 09 16:35:09.631814 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.631780    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-dshm\") pod \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") "
Sep 09 16:35:09.631995 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.631838    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-tmp-dir\") pod \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") "
Sep 09 16:35:09.631995 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.631865    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbmst\" (UniqueName: \"kubernetes.io/projected/94bfeea1-439f-4d47-b63e-0e7ba87bced1-kube-api-access-tbmst\") pod \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\" (UID: \"94bfeea1-439f-4d47-b63e-0e7ba87bced1\") "
Sep 09 16:35:09.631995 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.631949    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-home" (OuterVolumeSpecName: "home") pod "94bfeea1-439f-4d47-b63e-0e7ba87bced1" (UID: "94bfeea1-439f-4d47-b63e-0e7ba87bced1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:09.632149 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.632068    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-model-cache" (OuterVolumeSpecName: "model-cache") pod "94bfeea1-439f-4d47-b63e-0e7ba87bced1" (UID: "94bfeea1-439f-4d47-b63e-0e7ba87bced1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:09.632210 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.632166    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94bfeea1-439f-4d47-b63e-0e7ba87bced1" (UID: "94bfeea1-439f-4d47-b63e-0e7ba87bced1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:09.632268 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.632239    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:09.632268 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.632256    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:09.632349 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.632269    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:09.633931 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.633906    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94bfeea1-439f-4d47-b63e-0e7ba87bced1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94bfeea1-439f-4d47-b63e-0e7ba87bced1" (UID: "94bfeea1-439f-4d47-b63e-0e7ba87bced1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:35:09.634027 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.633983    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-dshm" (OuterVolumeSpecName: "dshm") pod "94bfeea1-439f-4d47-b63e-0e7ba87bced1" (UID: "94bfeea1-439f-4d47-b63e-0e7ba87bced1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:09.634074 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.634028    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bfeea1-439f-4d47-b63e-0e7ba87bced1-kube-api-access-tbmst" (OuterVolumeSpecName: "kube-api-access-tbmst") pod "94bfeea1-439f-4d47-b63e-0e7ba87bced1" (UID: "94bfeea1-439f-4d47-b63e-0e7ba87bced1"). InnerVolumeSpecName "kube-api-access-tbmst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:35:09.732981 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.732957    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94bfeea1-439f-4d47-b63e-0e7ba87bced1-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:09.732981 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.732982    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94bfeea1-439f-4d47-b63e-0e7ba87bced1-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:09.733165 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.732992    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbmst\" (UniqueName: \"kubernetes.io/projected/94bfeea1-439f-4d47-b63e-0e7ba87bced1-kube-api-access-tbmst\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:09.856016 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.855977    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p" event={"ID":"bdbc0851-7ea9-41e3-b8db-c39c446300da","Type":"ContainerStarted","Data":"cefebdc9927040430cd03f2bf2989fc16678f395e1cbf46ff49ac5f34d08fb38"}
Sep 09 16:35:09.856016 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.856028    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p" event={"ID":"bdbc0851-7ea9-41e3-b8db-c39c446300da","Type":"ContainerStarted","Data":"8bc8d1f5b36efafa34d448e6c6a0d1c745a352c407157dd245d07231439a5d54"}
Sep 09 16:35:09.857266 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.857235    2576 generic.go:356] "Generic (PLEG): container finished" podID="94bfeea1-439f-4d47-b63e-0e7ba87bced1" containerID="4735bcfc7eaad35566b1dac6da55a9bfe87770097f0185adde1611b4ddb77da4" exitCode=0
Sep 09 16:35:09.857399 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.857268    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"
Sep 09 16:35:09.857399 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.857313    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998" event={"ID":"94bfeea1-439f-4d47-b63e-0e7ba87bced1","Type":"ContainerDied","Data":"4735bcfc7eaad35566b1dac6da55a9bfe87770097f0185adde1611b4ddb77da4"}
Sep 09 16:35:09.857399 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.857350    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998" event={"ID":"94bfeea1-439f-4d47-b63e-0e7ba87bced1","Type":"ContainerDied","Data":"1f7c29cee8f76e16efafae1f0a1dbbba3ef349a9c04910685b87274c5bdb21cd"}
Sep 09 16:35:09.857399 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.857366    2576 scope.go:117] "RemoveContainer" containerID="4735bcfc7eaad35566b1dac6da55a9bfe87770097f0185adde1611b4ddb77da4"
Sep 09 16:35:09.868238 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.868219    2576 scope.go:117] "RemoveContainer" containerID="4735bcfc7eaad35566b1dac6da55a9bfe87770097f0185adde1611b4ddb77da4"
Sep 09 16:35:09.868582 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:35:09.868558    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4735bcfc7eaad35566b1dac6da55a9bfe87770097f0185adde1611b4ddb77da4\": container with ID starting with 4735bcfc7eaad35566b1dac6da55a9bfe87770097f0185adde1611b4ddb77da4 not found: ID does not exist" containerID="4735bcfc7eaad35566b1dac6da55a9bfe87770097f0185adde1611b4ddb77da4"
Sep 09 16:35:09.868708 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.868591    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4735bcfc7eaad35566b1dac6da55a9bfe87770097f0185adde1611b4ddb77da4"} err="failed to get container status \"4735bcfc7eaad35566b1dac6da55a9bfe87770097f0185adde1611b4ddb77da4\": rpc error: code = NotFound desc = could not find container \"4735bcfc7eaad35566b1dac6da55a9bfe87770097f0185adde1611b4ddb77da4\": container with ID starting with 4735bcfc7eaad35566b1dac6da55a9bfe87770097f0185adde1611b4ddb77da4 not found: ID does not exist"
Sep 09 16:35:09.872477 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.872438    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p" podStartSLOduration=1.872425368 podStartE2EDuration="1.872425368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:35:09.871928256 +0000 UTC m=+1585.807546699" watchObservedRunningTime="2026-09-09 16:35:09.872425368 +0000 UTC m=+1585.808043833"
Sep 09 16:35:09.885832 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.885809    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"]
Sep 09 16:35:09.889125 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:09.889105    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/sub-test-model-041bf1-kserve-8645cbb6c5-m5998"]
Sep 09 16:35:10.355919 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.355821    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg" podUID="9da41616-bd38-4dac-8892-9b17f65d94cf" containerName="main" containerID="cri-o://a33546564888041a79b51ce4e0a2c24532d66b2147f5940e72bb4f8938d3a4a3" gracePeriod=45
Sep 09 16:35:10.554008 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.553974    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94bfeea1-439f-4d47-b63e-0e7ba87bced1" path="/var/lib/kubelet/pods/94bfeea1-439f-4d47-b63e-0e7ba87bced1/volumes"
Sep 09 16:35:10.615763 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.613849    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:35:10.643369 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.641997    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-tmp-dir\") pod \"9da41616-bd38-4dac-8892-9b17f65d94cf\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") "
Sep 09 16:35:10.643369 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.642042    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da41616-bd38-4dac-8892-9b17f65d94cf-tls-certs\") pod \"9da41616-bd38-4dac-8892-9b17f65d94cf\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") "
Sep 09 16:35:10.643369 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.642070    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-home\") pod \"9da41616-bd38-4dac-8892-9b17f65d94cf\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") "
Sep 09 16:35:10.643369 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.642151    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-model-cache\") pod \"9da41616-bd38-4dac-8892-9b17f65d94cf\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") "
Sep 09 16:35:10.643369 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.642219    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-667gk\" (UniqueName: \"kubernetes.io/projected/9da41616-bd38-4dac-8892-9b17f65d94cf-kube-api-access-667gk\") pod \"9da41616-bd38-4dac-8892-9b17f65d94cf\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") "
Sep 09 16:35:10.643369 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.642242    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-dshm\") pod \"9da41616-bd38-4dac-8892-9b17f65d94cf\" (UID: \"9da41616-bd38-4dac-8892-9b17f65d94cf\") "
Sep 09 16:35:10.643369 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.642940    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9da41616-bd38-4dac-8892-9b17f65d94cf" (UID: "9da41616-bd38-4dac-8892-9b17f65d94cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:10.643902 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.643589    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-home" (OuterVolumeSpecName: "home") pod "9da41616-bd38-4dac-8892-9b17f65d94cf" (UID: "9da41616-bd38-4dac-8892-9b17f65d94cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:10.644251 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.644225    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "9da41616-bd38-4dac-8892-9b17f65d94cf" (UID: "9da41616-bd38-4dac-8892-9b17f65d94cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:10.651570 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.648052    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da41616-bd38-4dac-8892-9b17f65d94cf-kube-api-access-667gk" (OuterVolumeSpecName: "kube-api-access-667gk") pod "9da41616-bd38-4dac-8892-9b17f65d94cf" (UID: "9da41616-bd38-4dac-8892-9b17f65d94cf"). InnerVolumeSpecName "kube-api-access-667gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:35:10.652241 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.651788    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da41616-bd38-4dac-8892-9b17f65d94cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9da41616-bd38-4dac-8892-9b17f65d94cf" (UID: "9da41616-bd38-4dac-8892-9b17f65d94cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:35:10.655091 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.654773    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-dshm" (OuterVolumeSpecName: "dshm") pod "9da41616-bd38-4dac-8892-9b17f65d94cf" (UID: "9da41616-bd38-4dac-8892-9b17f65d94cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:10.742983 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.742949    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:10.742983 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.742978    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-667gk\" (UniqueName: \"kubernetes.io/projected/9da41616-bd38-4dac-8892-9b17f65d94cf-kube-api-access-667gk\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:10.742983 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.742991    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:10.743216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.743002    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:10.743216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.743010    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da41616-bd38-4dac-8892-9b17f65d94cf-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:10.743216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.743018    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da41616-bd38-4dac-8892-9b17f65d94cf-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:10.863457 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.863420    2576 generic.go:356] "Generic (PLEG): container finished" podID="9da41616-bd38-4dac-8892-9b17f65d94cf" containerID="a33546564888041a79b51ce4e0a2c24532d66b2147f5940e72bb4f8938d3a4a3" exitCode=0
Sep 09 16:35:10.863949 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.863464    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"
Sep 09 16:35:10.863949 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.863505    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg" event={"ID":"9da41616-bd38-4dac-8892-9b17f65d94cf","Type":"ContainerDied","Data":"a33546564888041a79b51ce4e0a2c24532d66b2147f5940e72bb4f8938d3a4a3"}
Sep 09 16:35:10.863949 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.863550    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg" event={"ID":"9da41616-bd38-4dac-8892-9b17f65d94cf","Type":"ContainerDied","Data":"8c82e36816788873a0442494a9be0ba3c94030d84313e812e2270a485e4fc559"}
Sep 09 16:35:10.863949 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.863575    2576 scope.go:117] "RemoveContainer" containerID="a33546564888041a79b51ce4e0a2c24532d66b2147f5940e72bb4f8938d3a4a3"
Sep 09 16:35:10.872273 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.872256    2576 scope.go:117] "RemoveContainer" containerID="a33546564888041a79b51ce4e0a2c24532d66b2147f5940e72bb4f8938d3a4a3"
Sep 09 16:35:10.872529 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:35:10.872511    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33546564888041a79b51ce4e0a2c24532d66b2147f5940e72bb4f8938d3a4a3\": container with ID starting with a33546564888041a79b51ce4e0a2c24532d66b2147f5940e72bb4f8938d3a4a3 not found: ID does not exist" containerID="a33546564888041a79b51ce4e0a2c24532d66b2147f5940e72bb4f8938d3a4a3"
Sep 09 16:35:10.872574 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.872538    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33546564888041a79b51ce4e0a2c24532d66b2147f5940e72bb4f8938d3a4a3"} err="failed to get container status \"a33546564888041a79b51ce4e0a2c24532d66b2147f5940e72bb4f8938d3a4a3\": rpc error: code = NotFound desc = could not find container \"a33546564888041a79b51ce4e0a2c24532d66b2147f5940e72bb4f8938d3a4a3\": container with ID starting with a33546564888041a79b51ce4e0a2c24532d66b2147f5940e72bb4f8938d3a4a3 not found: ID does not exist"
Sep 09 16:35:10.884025 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.883997    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"]
Sep 09 16:35:10.885721 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:10.885687    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/sub-test-model-9520a4-kserve-7fbc8d74d9-sqjlg"]
Sep 09 16:35:12.553223 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:12.553187    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9da41616-bd38-4dac-8892-9b17f65d94cf" path="/var/lib/kubelet/pods/9da41616-bd38-4dac-8892-9b17f65d94cf/volumes"
Sep 09 16:35:13.830568 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:13.830537    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"
Sep 09 16:35:19.119669 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:19.119617    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:19.119669 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:19.119679    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:19.132300 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:19.132278    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:19.919132 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:19.919104    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:35:28.403991 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.403953    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv"]
Sep 09 16:35:28.404353 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.404280    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9da41616-bd38-4dac-8892-9b17f65d94cf" containerName="main"
Sep 09 16:35:28.404353 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.404290    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9da41616-bd38-4dac-8892-9b17f65d94cf" containerName="main"
Sep 09 16:35:28.404353 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.404308    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94bfeea1-439f-4d47-b63e-0e7ba87bced1" containerName="main"
Sep 09 16:35:28.404353 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.404313    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="94bfeea1-439f-4d47-b63e-0e7ba87bced1" containerName="main"
Sep 09 16:35:28.404486 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.404375    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="94bfeea1-439f-4d47-b63e-0e7ba87bced1" containerName="main"
Sep 09 16:35:28.404486 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.404384    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9da41616-bd38-4dac-8892-9b17f65d94cf" containerName="main"
Sep 09 16:35:28.407390 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.407372    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv"
Sep 09 16:35:28.409674 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.409654    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:35:28.409792 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.409774    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxghs\""
Sep 09 16:35:28.413680 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.413658    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv"]
Sep 09 16:35:28.501391 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.501358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv\" (UID: \"fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv"
Sep 09 16:35:28.602393 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.602364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv\" (UID: \"fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv"
Sep 09 16:35:28.602984 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.602966    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv\" (UID: \"fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv"
Sep 09 16:35:28.720034 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.719959    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv"
Sep 09 16:35:28.987144 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:28.987118    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv"]
Sep 09 16:35:28.990283 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:35:28.990257    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc7ccc4_3b39_4e0b_b5f9_ccc7bef35213.slice/crio-8bf0c795afb33dce1fbc5701781b8105c17819f20e56a066ed5563e30fe011a8 WatchSource:0}: Error finding container 8bf0c795afb33dce1fbc5701781b8105c17819f20e56a066ed5563e30fe011a8: Status 404 returned error can't find the container with id 8bf0c795afb33dce1fbc5701781b8105c17819f20e56a066ed5563e30fe011a8
Sep 09 16:35:29.955110 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:29.955024    2576 generic.go:356] "Generic (PLEG): container finished" podID="fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213" containerID="234841c7301c2d734bef0050663d5331ed3e2609d6ee009648ccdd8939e094f9" exitCode=0
Sep 09 16:35:29.955450 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:29.955115    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv" event={"ID":"fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213","Type":"ContainerDied","Data":"234841c7301c2d734bef0050663d5331ed3e2609d6ee009648ccdd8939e094f9"}
Sep 09 16:35:29.955450 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:29.955191    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv" event={"ID":"fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213","Type":"ContainerStarted","Data":"8bf0c795afb33dce1fbc5701781b8105c17819f20e56a066ed5563e30fe011a8"}
Sep 09 16:35:31.083887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:31.083863    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv"
Sep 09 16:35:31.226922 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:31.226895    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213-maas-api-service-ca\") pod \"fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213\" (UID: \"fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213\") "
Sep 09 16:35:31.227230 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:31.227208    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213" (UID: "fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:35:31.328238 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:31.328211    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213-maas-api-service-ca\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:31.965136 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:31.965101    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv" event={"ID":"fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213","Type":"ContainerDied","Data":"8bf0c795afb33dce1fbc5701781b8105c17819f20e56a066ed5563e30fe011a8"}
Sep 09 16:35:31.965303 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:31.965145    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bf0c795afb33dce1fbc5701781b8105c17819f20e56a066ed5563e30fe011a8"
Sep 09 16:35:31.965303 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:31.965109    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv"
Sep 09 16:35:43.906319 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:43.906236    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"]
Sep 09 16:35:43.906781 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:43.906501    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf" podUID="373194db-188d-4dcc-90b9-10f7941069d9" containerName="payload-processing" containerID="cri-o://14a4d245c33f6e30379f5bfacbc0b89c3dd597f17f90bb19e52c516faaf39ba0" gracePeriod=30
Sep 09 16:35:44.074934 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:44.074757    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv"]
Sep 09 16:35:44.077422 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:44.077395    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d8327-8c6xv"]
Sep 09 16:35:44.165024 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:44.164970    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"
Sep 09 16:35:44.338497 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:44.338468    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373194db-188d-4dcc-90b9-10f7941069d9-plugins-config-volume\") pod \"373194db-188d-4dcc-90b9-10f7941069d9\" (UID: \"373194db-188d-4dcc-90b9-10f7941069d9\") "
Sep 09 16:35:44.338655 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:44.338527    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxj55\" (UniqueName: \"kubernetes.io/projected/373194db-188d-4dcc-90b9-10f7941069d9-kube-api-access-qxj55\") pod \"373194db-188d-4dcc-90b9-10f7941069d9\" (UID: \"373194db-188d-4dcc-90b9-10f7941069d9\") "
Sep 09 16:35:44.338936 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:44.338910    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373194db-188d-4dcc-90b9-10f7941069d9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "373194db-188d-4dcc-90b9-10f7941069d9" (UID: "373194db-188d-4dcc-90b9-10f7941069d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:35:44.340730 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:44.340709    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373194db-188d-4dcc-90b9-10f7941069d9-kube-api-access-qxj55" (OuterVolumeSpecName: "kube-api-access-qxj55") pod "373194db-188d-4dcc-90b9-10f7941069d9" (UID: "373194db-188d-4dcc-90b9-10f7941069d9"). InnerVolumeSpecName "kube-api-access-qxj55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:35:44.439851 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:44.439790    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxj55\" (UniqueName: \"kubernetes.io/projected/373194db-188d-4dcc-90b9-10f7941069d9-kube-api-access-qxj55\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:44.439851 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:44.439811    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373194db-188d-4dcc-90b9-10f7941069d9-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:44.553214 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:44.553177    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213" path="/var/lib/kubelet/pods/fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213/volumes"
Sep 09 16:35:45.024720 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:45.024665    2576 generic.go:356] "Generic (PLEG): container finished" podID="373194db-188d-4dcc-90b9-10f7941069d9" containerID="14a4d245c33f6e30379f5bfacbc0b89c3dd597f17f90bb19e52c516faaf39ba0" exitCode=0
Sep 09 16:35:45.025153 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:45.024753    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"
Sep 09 16:35:45.025153 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:45.024754    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf" event={"ID":"373194db-188d-4dcc-90b9-10f7941069d9","Type":"ContainerDied","Data":"14a4d245c33f6e30379f5bfacbc0b89c3dd597f17f90bb19e52c516faaf39ba0"}
Sep 09 16:35:45.025153 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:45.024810    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf" event={"ID":"373194db-188d-4dcc-90b9-10f7941069d9","Type":"ContainerDied","Data":"effdcd987f917abd60b1b2351f3216ed68afcf865d68d529c081a95050f85357"}
Sep 09 16:35:45.025153 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:45.024860    2576 scope.go:117] "RemoveContainer" containerID="14a4d245c33f6e30379f5bfacbc0b89c3dd597f17f90bb19e52c516faaf39ba0"
Sep 09 16:35:45.033195 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:45.033180    2576 scope.go:117] "RemoveContainer" containerID="14a4d245c33f6e30379f5bfacbc0b89c3dd597f17f90bb19e52c516faaf39ba0"
Sep 09 16:35:45.033456 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:35:45.033439    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a4d245c33f6e30379f5bfacbc0b89c3dd597f17f90bb19e52c516faaf39ba0\": container with ID starting with 14a4d245c33f6e30379f5bfacbc0b89c3dd597f17f90bb19e52c516faaf39ba0 not found: ID does not exist" containerID="14a4d245c33f6e30379f5bfacbc0b89c3dd597f17f90bb19e52c516faaf39ba0"
Sep 09 16:35:45.033501 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:45.033467    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a4d245c33f6e30379f5bfacbc0b89c3dd597f17f90bb19e52c516faaf39ba0"} err="failed to get container status \"14a4d245c33f6e30379f5bfacbc0b89c3dd597f17f90bb19e52c516faaf39ba0\": rpc error: code = NotFound desc = could not find container \"14a4d245c33f6e30379f5bfacbc0b89c3dd597f17f90bb19e52c516faaf39ba0\": container with ID starting with 14a4d245c33f6e30379f5bfacbc0b89c3dd597f17f90bb19e52c516faaf39ba0 not found: ID does not exist"
Sep 09 16:35:45.041112 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:45.041091    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"]
Sep 09 16:35:45.047220 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:45.047201    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d8327-69c6cbcf7c-b2mzf"]
Sep 09 16:35:46.553759 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:46.553724    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="373194db-188d-4dcc-90b9-10f7941069d9" path="/var/lib/kubelet/pods/373194db-188d-4dcc-90b9-10f7941069d9/volumes"
Sep 09 16:35:50.888728 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:50.888681    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6"]
Sep 09 16:35:50.889188 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:50.889008    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213" containerName="revoke-keys"
Sep 09 16:35:50.889188 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:50.889019    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213" containerName="revoke-keys"
Sep 09 16:35:50.889188 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:50.889049    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373194db-188d-4dcc-90b9-10f7941069d9" containerName="payload-processing"
Sep 09 16:35:50.889188 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:50.889055    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="373194db-188d-4dcc-90b9-10f7941069d9" containerName="payload-processing"
Sep 09 16:35:50.889188 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:50.889108    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcc7ccc4-3b39-4e0b-b5f9-ccc7bef35213" containerName="revoke-keys"
Sep 09 16:35:50.889188 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:50.889118    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="373194db-188d-4dcc-90b9-10f7941069d9" containerName="payload-processing"
Sep 09 16:35:50.893536 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:50.893521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6"
Sep 09 16:35:50.895977 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:50.895959    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxghs\""
Sep 09 16:35:50.896085 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:50.896005    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:35:50.899344 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:50.899324    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6"]
Sep 09 16:35:50.994789 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:50.994748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50acdaf9-076b-4604-92d6-fde0c3753b51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6\" (UID: \"50acdaf9-076b-4604-92d6-fde0c3753b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6"
Sep 09 16:35:51.096165 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:51.096132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50acdaf9-076b-4604-92d6-fde0c3753b51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6\" (UID: \"50acdaf9-076b-4604-92d6-fde0c3753b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6"
Sep 09 16:35:51.096679 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:51.096658    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50acdaf9-076b-4604-92d6-fde0c3753b51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6\" (UID: \"50acdaf9-076b-4604-92d6-fde0c3753b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6"
Sep 09 16:35:51.204445 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:51.204362    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6"
Sep 09 16:35:51.471479 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:51.471456    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6"]
Sep 09 16:35:51.473741 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:35:51.473681    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50acdaf9_076b_4604_92d6_fde0c3753b51.slice/crio-4d1bbf458378cfcce6dbfa1f0bbd22ffa3588ab357d10112df9a3606c50b1c56 WatchSource:0}: Error finding container 4d1bbf458378cfcce6dbfa1f0bbd22ffa3588ab357d10112df9a3606c50b1c56: Status 404 returned error can't find the container with id 4d1bbf458378cfcce6dbfa1f0bbd22ffa3588ab357d10112df9a3606c50b1c56
Sep 09 16:35:52.060627 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:52.060540    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6" event={"ID":"50acdaf9-076b-4604-92d6-fde0c3753b51","Type":"ContainerStarted","Data":"a1ada3f591a4e3b6d68d7e96d623ba7544022af33669266041f6e92b364894f0"}
Sep 09 16:35:52.060627 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:52.060589    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6" event={"ID":"50acdaf9-076b-4604-92d6-fde0c3753b51","Type":"ContainerStarted","Data":"4d1bbf458378cfcce6dbfa1f0bbd22ffa3588ab357d10112df9a3606c50b1c56"}
Sep 09 16:35:52.075035 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:52.074988    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6" podStartSLOduration=2.074970722 podStartE2EDuration="2.074970722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:35:52.074878421 +0000 UTC m=+1628.010496864" watchObservedRunningTime="2026-09-09 16:35:52.074970722 +0000 UTC m=+1628.010589139"
Sep 09 16:35:53.066267 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:53.066233    2576 generic.go:356] "Generic (PLEG): container finished" podID="50acdaf9-076b-4604-92d6-fde0c3753b51" containerID="a1ada3f591a4e3b6d68d7e96d623ba7544022af33669266041f6e92b364894f0" exitCode=0
Sep 09 16:35:53.066267 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:53.066268    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6" event={"ID":"50acdaf9-076b-4604-92d6-fde0c3753b51","Type":"ContainerDied","Data":"a1ada3f591a4e3b6d68d7e96d623ba7544022af33669266041f6e92b364894f0"}
Sep 09 16:35:54.195245 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:54.195220    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6"
Sep 09 16:35:54.221354 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:54.221321    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50acdaf9-076b-4604-92d6-fde0c3753b51-maas-api-service-ca\") pod \"50acdaf9-076b-4604-92d6-fde0c3753b51\" (UID: \"50acdaf9-076b-4604-92d6-fde0c3753b51\") "
Sep 09 16:35:54.221691 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:54.221666    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50acdaf9-076b-4604-92d6-fde0c3753b51-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50acdaf9-076b-4604-92d6-fde0c3753b51" (UID: "50acdaf9-076b-4604-92d6-fde0c3753b51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:35:54.323006 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:54.322922    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50acdaf9-076b-4604-92d6-fde0c3753b51-maas-api-service-ca\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:35:55.076753 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:55.076725    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6"
Sep 09 16:35:55.076929 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:55.076717    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6" event={"ID":"50acdaf9-076b-4604-92d6-fde0c3753b51","Type":"ContainerDied","Data":"4d1bbf458378cfcce6dbfa1f0bbd22ffa3588ab357d10112df9a3606c50b1c56"}
Sep 09 16:35:55.076929 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:55.076840    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d1bbf458378cfcce6dbfa1f0bbd22ffa3588ab357d10112df9a3606c50b1c56"
Sep 09 16:35:58.805868 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.805824    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"]
Sep 09 16:35:58.806339 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.806323    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50acdaf9-076b-4604-92d6-fde0c3753b51" containerName="revoke-keys"
Sep 09 16:35:58.806402 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.806342    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50acdaf9-076b-4604-92d6-fde0c3753b51" containerName="revoke-keys"
Sep 09 16:35:58.806458 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.806426    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50acdaf9-076b-4604-92d6-fde0c3753b51" containerName="revoke-keys"
Sep 09 16:35:58.809532 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.809511    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.812016 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.811996    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9520a4\"/\"openshift-service-ca.crt\""
Sep 09 16:35:58.813018 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.813002    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9520a4\"/\"kube-root-ca.crt\""
Sep 09 16:35:58.813018 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.813014    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9520a4\"/\"rate-test-model-9520a4-kserve-self-signed-certs\""
Sep 09 16:35:58.813216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.813094    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9520a4\"/\"default-dockercfg-qlph4\""
Sep 09 16:35:58.819170 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.819149    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"]
Sep 09 16:35:58.862565 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.862535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-tmp-dir\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.862738 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.862591    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-model-cache\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.862738 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.862720    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-home\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.862845 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.862766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-dshm\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.862845 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.862786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4dwz\" (UniqueName: \"kubernetes.io/projected/9e2db2a1-72ff-4750-bab2-946632df5c71-kube-api-access-p4dwz\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.862927 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.862874    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2db2a1-72ff-4750-bab2-946632df5c71-tls-certs\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.964014 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.963982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-model-cache\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.964205 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.964037    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-home\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.964205 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.964165    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-dshm\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.964335 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.964201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4dwz\" (UniqueName: \"kubernetes.io/projected/9e2db2a1-72ff-4750-bab2-946632df5c71-kube-api-access-p4dwz\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.964335 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.964257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2db2a1-72ff-4750-bab2-946632df5c71-tls-certs\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.964335 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.964303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-tmp-dir\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.964493 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.964400    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-home\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.964493 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.964425    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-model-cache\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.964645 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.964626    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-tmp-dir\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.966428 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.966408    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-dshm\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.966733 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.966686    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2db2a1-72ff-4750-bab2-946632df5c71-tls-certs\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:58.971421 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:58.971400    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4dwz\" (UniqueName: \"kubernetes.io/projected/9e2db2a1-72ff-4750-bab2-946632df5c71-kube-api-access-p4dwz\") pod \"rate-test-model-9520a4-kserve-6b4fd94599-vf2t9\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") " pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:59.119518 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:59.119432    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:35:59.396574 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:35:59.396497    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"]
Sep 09 16:35:59.399324 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:35:59.399286    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e2db2a1_72ff_4750_bab2_946632df5c71.slice/crio-a42128d23b60eac30c8785581a8ac076b544cb317a4975c7f0a3baeedcc4d4c9 WatchSource:0}: Error finding container a42128d23b60eac30c8785581a8ac076b544cb317a4975c7f0a3baeedcc4d4c9: Status 404 returned error can't find the container with id a42128d23b60eac30c8785581a8ac076b544cb317a4975c7f0a3baeedcc4d4c9
Sep 09 16:36:00.107767 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:00.107731    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9" event={"ID":"9e2db2a1-72ff-4750-bab2-946632df5c71","Type":"ContainerStarted","Data":"dff77c0f380048bc5c63f4aeb98216a6cc0de963e55fdfcfa0775567ca057f94"}
Sep 09 16:36:00.108164 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:00.107782    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9" event={"ID":"9e2db2a1-72ff-4750-bab2-946632df5c71","Type":"ContainerStarted","Data":"a42128d23b60eac30c8785581a8ac076b544cb317a4975c7f0a3baeedcc4d4c9"}
Sep 09 16:36:00.123157 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:00.123115    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9" podStartSLOduration=2.123099952 podStartE2EDuration="2.123099952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:36:00.122798199 +0000 UTC m=+1636.058416614" watchObservedRunningTime="2026-09-09 16:36:00.123099952 +0000 UTC m=+1636.058718368"
Sep 09 16:36:06.400811 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:06.400775    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"]
Sep 09 16:36:06.401274 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:06.401117    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw" podUID="985e28bd-b0f2-4f3a-a532-ea24bbb04e05" containerName="payload-pre-processing" containerID="cri-o://778db26d7d3b9bc287b15584d0454e986fb412ece912406476aa867dfd7340e8" gracePeriod=30
Sep 09 16:36:06.554884 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:06.554742    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6"]
Sep 09 16:36:06.556396 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:06.556370    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab78ef-4f2x6"]
Sep 09 16:36:06.647022 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:06.646999    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"
Sep 09 16:36:06.736215 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:06.736167    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985e28bd-b0f2-4f3a-a532-ea24bbb04e05-plugins-config-volume\") pod \"985e28bd-b0f2-4f3a-a532-ea24bbb04e05\" (UID: \"985e28bd-b0f2-4f3a-a532-ea24bbb04e05\") "
Sep 09 16:36:06.736417 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:06.736297    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn62z\" (UniqueName: \"kubernetes.io/projected/985e28bd-b0f2-4f3a-a532-ea24bbb04e05-kube-api-access-dn62z\") pod \"985e28bd-b0f2-4f3a-a532-ea24bbb04e05\" (UID: \"985e28bd-b0f2-4f3a-a532-ea24bbb04e05\") "
Sep 09 16:36:06.736597 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:06.736567    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985e28bd-b0f2-4f3a-a532-ea24bbb04e05-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "985e28bd-b0f2-4f3a-a532-ea24bbb04e05" (UID: "985e28bd-b0f2-4f3a-a532-ea24bbb04e05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:36:06.736735 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:06.736688    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985e28bd-b0f2-4f3a-a532-ea24bbb04e05-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:36:06.738522 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:06.738494    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985e28bd-b0f2-4f3a-a532-ea24bbb04e05-kube-api-access-dn62z" (OuterVolumeSpecName: "kube-api-access-dn62z") pod "985e28bd-b0f2-4f3a-a532-ea24bbb04e05" (UID: "985e28bd-b0f2-4f3a-a532-ea24bbb04e05"). InnerVolumeSpecName "kube-api-access-dn62z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:36:06.838111 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:06.838052    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn62z\" (UniqueName: \"kubernetes.io/projected/985e28bd-b0f2-4f3a-a532-ea24bbb04e05-kube-api-access-dn62z\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:36:07.148117 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:07.148025    2576 generic.go:356] "Generic (PLEG): container finished" podID="985e28bd-b0f2-4f3a-a532-ea24bbb04e05" containerID="778db26d7d3b9bc287b15584d0454e986fb412ece912406476aa867dfd7340e8" exitCode=0
Sep 09 16:36:07.148117 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:07.148085    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"
Sep 09 16:36:07.148305 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:07.148106    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw" event={"ID":"985e28bd-b0f2-4f3a-a532-ea24bbb04e05","Type":"ContainerDied","Data":"778db26d7d3b9bc287b15584d0454e986fb412ece912406476aa867dfd7340e8"}
Sep 09 16:36:07.148305 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:07.148193    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw" event={"ID":"985e28bd-b0f2-4f3a-a532-ea24bbb04e05","Type":"ContainerDied","Data":"a9421a7759276a5dcb5dc8849fe840f523fea254f61a83780e728730a286c6e9"}
Sep 09 16:36:07.148305 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:07.148223    2576 scope.go:117] "RemoveContainer" containerID="778db26d7d3b9bc287b15584d0454e986fb412ece912406476aa867dfd7340e8"
Sep 09 16:36:07.158471 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:07.158445    2576 scope.go:117] "RemoveContainer" containerID="778db26d7d3b9bc287b15584d0454e986fb412ece912406476aa867dfd7340e8"
Sep 09 16:36:07.158824 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:36:07.158796    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778db26d7d3b9bc287b15584d0454e986fb412ece912406476aa867dfd7340e8\": container with ID starting with 778db26d7d3b9bc287b15584d0454e986fb412ece912406476aa867dfd7340e8 not found: ID does not exist" containerID="778db26d7d3b9bc287b15584d0454e986fb412ece912406476aa867dfd7340e8"
Sep 09 16:36:07.158934 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:07.158835    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778db26d7d3b9bc287b15584d0454e986fb412ece912406476aa867dfd7340e8"} err="failed to get container status \"778db26d7d3b9bc287b15584d0454e986fb412ece912406476aa867dfd7340e8\": rpc error: code = NotFound desc = could not find container \"778db26d7d3b9bc287b15584d0454e986fb412ece912406476aa867dfd7340e8\": container with ID starting with 778db26d7d3b9bc287b15584d0454e986fb412ece912406476aa867dfd7340e8 not found: ID does not exist"
Sep 09 16:36:07.168031 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:07.168007    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"]
Sep 09 16:36:07.173923 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:07.173894    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab78ef-6cf654d54b-49xrw"]
Sep 09 16:36:08.553280 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:08.553247    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50acdaf9-076b-4604-92d6-fde0c3753b51" path="/var/lib/kubelet/pods/50acdaf9-076b-4604-92d6-fde0c3753b51/volumes"
Sep 09 16:36:08.553644 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:08.553586    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="985e28bd-b0f2-4f3a-a532-ea24bbb04e05" path="/var/lib/kubelet/pods/985e28bd-b0f2-4f3a-a532-ea24bbb04e05/volumes"
Sep 09 16:36:09.119620 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:09.119587    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:36:09.119852 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:09.119645    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:36:09.131861 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:09.131836    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:36:09.170331 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:09.170304    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:36:13.724229 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:13.724200    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w"]
Sep 09 16:36:13.724577 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:13.724523    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="985e28bd-b0f2-4f3a-a532-ea24bbb04e05" containerName="payload-pre-processing"
Sep 09 16:36:13.724577 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:13.724533    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="985e28bd-b0f2-4f3a-a532-ea24bbb04e05" containerName="payload-pre-processing"
Sep 09 16:36:13.724654 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:13.724595    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="985e28bd-b0f2-4f3a-a532-ea24bbb04e05" containerName="payload-pre-processing"
Sep 09 16:36:13.728815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:13.728796    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w"
Sep 09 16:36:13.731280 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:13.731249    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:36:13.731400 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:13.731335    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxghs\""
Sep 09 16:36:13.734733 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:13.734689    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w"]
Sep 09 16:36:13.797601 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:13.797561    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbafabb-8232-41f8-bbaf-f0c4b0e137c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w\" (UID: \"4fbafabb-8232-41f8-bbaf-f0c4b0e137c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w"
Sep 09 16:36:13.898802 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:13.898763    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbafabb-8232-41f8-bbaf-f0c4b0e137c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w\" (UID: \"4fbafabb-8232-41f8-bbaf-f0c4b0e137c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w"
Sep 09 16:36:13.899398 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:13.899377    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbafabb-8232-41f8-bbaf-f0c4b0e137c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w\" (UID: \"4fbafabb-8232-41f8-bbaf-f0c4b0e137c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w"
Sep 09 16:36:14.040551 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:14.040454    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w"
Sep 09 16:36:14.317713 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:14.317673    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w"]
Sep 09 16:36:14.319492 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:36:14.319468    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbafabb_8232_41f8_bbaf_f0c4b0e137c6.slice/crio-6951c3cd216068fef0dd911b33a71865c214ea6beaf25092976076771037a771 WatchSource:0}: Error finding container 6951c3cd216068fef0dd911b33a71865c214ea6beaf25092976076771037a771: Status 404 returned error can't find the container with id 6951c3cd216068fef0dd911b33a71865c214ea6beaf25092976076771037a771
Sep 09 16:36:15.192223 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:15.192181    2576 generic.go:356] "Generic (PLEG): container finished" podID="4fbafabb-8232-41f8-bbaf-f0c4b0e137c6" containerID="13fc88215e4754eafb33fce907f7fadf9b149d0377cb6a33f1b292ee098540ba" exitCode=0
Sep 09 16:36:15.192223 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:15.192220    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w" event={"ID":"4fbafabb-8232-41f8-bbaf-f0c4b0e137c6","Type":"ContainerDied","Data":"13fc88215e4754eafb33fce907f7fadf9b149d0377cb6a33f1b292ee098540ba"}
Sep 09 16:36:15.192624 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:15.192272    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w" event={"ID":"4fbafabb-8232-41f8-bbaf-f0c4b0e137c6","Type":"ContainerStarted","Data":"6951c3cd216068fef0dd911b33a71865c214ea6beaf25092976076771037a771"}
Sep 09 16:36:16.326488 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:16.326463    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w"
Sep 09 16:36:16.419851 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:16.419818    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbafabb-8232-41f8-bbaf-f0c4b0e137c6-maas-api-service-ca\") pod \"4fbafabb-8232-41f8-bbaf-f0c4b0e137c6\" (UID: \"4fbafabb-8232-41f8-bbaf-f0c4b0e137c6\") "
Sep 09 16:36:16.420179 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:16.420157    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbafabb-8232-41f8-bbaf-f0c4b0e137c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4fbafabb-8232-41f8-bbaf-f0c4b0e137c6" (UID: "4fbafabb-8232-41f8-bbaf-f0c4b0e137c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:36:16.520902 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:16.520867    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbafabb-8232-41f8-bbaf-f0c4b0e137c6-maas-api-service-ca\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:36:17.201967 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:17.201934    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w" event={"ID":"4fbafabb-8232-41f8-bbaf-f0c4b0e137c6","Type":"ContainerDied","Data":"6951c3cd216068fef0dd911b33a71865c214ea6beaf25092976076771037a771"}
Sep 09 16:36:17.202119 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:17.201981    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6951c3cd216068fef0dd911b33a71865c214ea6beaf25092976076771037a771"
Sep 09 16:36:17.202119 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:17.201943    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w"
Sep 09 16:36:24.640460 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:24.640422    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"]
Sep 09 16:36:24.641053 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:24.640686    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl" podUID="eb09db09-f69c-4dad-a7dc-4ce7fe732674" containerName="payload-processing" containerID="cri-o://a6b091b86021cc4ba5afb643f5302e7ff751ffe12d85e4a9b3db35a703055bef" gracePeriod=30
Sep 09 16:36:24.647810 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:24.647784    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"]
Sep 09 16:36:24.648037 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:24.648017    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6" podUID="b4dd7c8c-f218-4f30-b6b6-087f4a1fa106" containerName="payload-pre-processing" containerID="cri-o://1963bbbaad78dceb80d0d6d493a4ead0a6ab7653f7e87a03e4e0cba317e28623" gracePeriod=30
Sep 09 16:36:24.796509 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:24.796477    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w"]
Sep 09 16:36:24.800918 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:24.800843    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-042f8b-ggk2w"]
Sep 09 16:36:24.918881 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:24.918856    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"
Sep 09 16:36:24.921827 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:24.921805    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"
Sep 09 16:36:25.102956 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.102924    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l4q6\" (UniqueName: \"kubernetes.io/projected/eb09db09-f69c-4dad-a7dc-4ce7fe732674-kube-api-access-7l4q6\") pod \"eb09db09-f69c-4dad-a7dc-4ce7fe732674\" (UID: \"eb09db09-f69c-4dad-a7dc-4ce7fe732674\") "
Sep 09 16:36:25.103133 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.103009    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb09db09-f69c-4dad-a7dc-4ce7fe732674-plugins-config-volume\") pod \"eb09db09-f69c-4dad-a7dc-4ce7fe732674\" (UID: \"eb09db09-f69c-4dad-a7dc-4ce7fe732674\") "
Sep 09 16:36:25.103133 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.103037    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4dd7c8c-f218-4f30-b6b6-087f4a1fa106-plugins-config-volume\") pod \"b4dd7c8c-f218-4f30-b6b6-087f4a1fa106\" (UID: \"b4dd7c8c-f218-4f30-b6b6-087f4a1fa106\") "
Sep 09 16:36:25.103133 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.103082    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vgxw\" (UniqueName: \"kubernetes.io/projected/b4dd7c8c-f218-4f30-b6b6-087f4a1fa106-kube-api-access-6vgxw\") pod \"b4dd7c8c-f218-4f30-b6b6-087f4a1fa106\" (UID: \"b4dd7c8c-f218-4f30-b6b6-087f4a1fa106\") "
Sep 09 16:36:25.103427 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.103402    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4dd7c8c-f218-4f30-b6b6-087f4a1fa106-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4dd7c8c-f218-4f30-b6b6-087f4a1fa106" (UID: "b4dd7c8c-f218-4f30-b6b6-087f4a1fa106"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:36:25.103427 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.103418    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb09db09-f69c-4dad-a7dc-4ce7fe732674-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb09db09-f69c-4dad-a7dc-4ce7fe732674" (UID: "eb09db09-f69c-4dad-a7dc-4ce7fe732674"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:36:25.105299 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.105278    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4dd7c8c-f218-4f30-b6b6-087f4a1fa106-kube-api-access-6vgxw" (OuterVolumeSpecName: "kube-api-access-6vgxw") pod "b4dd7c8c-f218-4f30-b6b6-087f4a1fa106" (UID: "b4dd7c8c-f218-4f30-b6b6-087f4a1fa106"). InnerVolumeSpecName "kube-api-access-6vgxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:36:25.105412 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.105396    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb09db09-f69c-4dad-a7dc-4ce7fe732674-kube-api-access-7l4q6" (OuterVolumeSpecName: "kube-api-access-7l4q6") pod "eb09db09-f69c-4dad-a7dc-4ce7fe732674" (UID: "eb09db09-f69c-4dad-a7dc-4ce7fe732674"). InnerVolumeSpecName "kube-api-access-7l4q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:36:25.204134 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.204043    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb09db09-f69c-4dad-a7dc-4ce7fe732674-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:36:25.204134 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.204074    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4dd7c8c-f218-4f30-b6b6-087f4a1fa106-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:36:25.204134 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.204087    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vgxw\" (UniqueName: \"kubernetes.io/projected/b4dd7c8c-f218-4f30-b6b6-087f4a1fa106-kube-api-access-6vgxw\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:36:25.204134 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.204096    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l4q6\" (UniqueName: \"kubernetes.io/projected/eb09db09-f69c-4dad-a7dc-4ce7fe732674-kube-api-access-7l4q6\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:36:25.241712 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.241659    2576 generic.go:356] "Generic (PLEG): container finished" podID="b4dd7c8c-f218-4f30-b6b6-087f4a1fa106" containerID="1963bbbaad78dceb80d0d6d493a4ead0a6ab7653f7e87a03e4e0cba317e28623" exitCode=0
Sep 09 16:36:25.241911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.241741    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"
Sep 09 16:36:25.241911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.241741    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6" event={"ID":"b4dd7c8c-f218-4f30-b6b6-087f4a1fa106","Type":"ContainerDied","Data":"1963bbbaad78dceb80d0d6d493a4ead0a6ab7653f7e87a03e4e0cba317e28623"}
Sep 09 16:36:25.241911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.241865    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6" event={"ID":"b4dd7c8c-f218-4f30-b6b6-087f4a1fa106","Type":"ContainerDied","Data":"4317e20ed29d5b56d5ee5aaa0e41fe5deaed8f2378d63713a4292df24926a82b"}
Sep 09 16:36:25.241911 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.241896    2576 scope.go:117] "RemoveContainer" containerID="1963bbbaad78dceb80d0d6d493a4ead0a6ab7653f7e87a03e4e0cba317e28623"
Sep 09 16:36:25.243198 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.243181    2576 generic.go:356] "Generic (PLEG): container finished" podID="eb09db09-f69c-4dad-a7dc-4ce7fe732674" containerID="a6b091b86021cc4ba5afb643f5302e7ff751ffe12d85e4a9b3db35a703055bef" exitCode=0
Sep 09 16:36:25.243307 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.243222    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"
Sep 09 16:36:25.243307 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.243240    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl" event={"ID":"eb09db09-f69c-4dad-a7dc-4ce7fe732674","Type":"ContainerDied","Data":"a6b091b86021cc4ba5afb643f5302e7ff751ffe12d85e4a9b3db35a703055bef"}
Sep 09 16:36:25.243307 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.243271    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl" event={"ID":"eb09db09-f69c-4dad-a7dc-4ce7fe732674","Type":"ContainerDied","Data":"e5ef886e082226397a4faeb1e25155f4f551c242db5d019cb59095ea701055a6"}
Sep 09 16:36:25.251639 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.251623    2576 scope.go:117] "RemoveContainer" containerID="1963bbbaad78dceb80d0d6d493a4ead0a6ab7653f7e87a03e4e0cba317e28623"
Sep 09 16:36:25.251965 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:36:25.251945    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1963bbbaad78dceb80d0d6d493a4ead0a6ab7653f7e87a03e4e0cba317e28623\": container with ID starting with 1963bbbaad78dceb80d0d6d493a4ead0a6ab7653f7e87a03e4e0cba317e28623 not found: ID does not exist" containerID="1963bbbaad78dceb80d0d6d493a4ead0a6ab7653f7e87a03e4e0cba317e28623"
Sep 09 16:36:25.252033 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.251974    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1963bbbaad78dceb80d0d6d493a4ead0a6ab7653f7e87a03e4e0cba317e28623"} err="failed to get container status \"1963bbbaad78dceb80d0d6d493a4ead0a6ab7653f7e87a03e4e0cba317e28623\": rpc error: code = NotFound desc = could not find container \"1963bbbaad78dceb80d0d6d493a4ead0a6ab7653f7e87a03e4e0cba317e28623\": container with ID starting with 1963bbbaad78dceb80d0d6d493a4ead0a6ab7653f7e87a03e4e0cba317e28623 not found: ID does not exist"
Sep 09 16:36:25.252033 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.251991    2576 scope.go:117] "RemoveContainer" containerID="a6b091b86021cc4ba5afb643f5302e7ff751ffe12d85e4a9b3db35a703055bef"
Sep 09 16:36:25.260233 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.260216    2576 scope.go:117] "RemoveContainer" containerID="a6b091b86021cc4ba5afb643f5302e7ff751ffe12d85e4a9b3db35a703055bef"
Sep 09 16:36:25.260495 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:36:25.260479    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b091b86021cc4ba5afb643f5302e7ff751ffe12d85e4a9b3db35a703055bef\": container with ID starting with a6b091b86021cc4ba5afb643f5302e7ff751ffe12d85e4a9b3db35a703055bef not found: ID does not exist" containerID="a6b091b86021cc4ba5afb643f5302e7ff751ffe12d85e4a9b3db35a703055bef"
Sep 09 16:36:25.260541 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.260502    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b091b86021cc4ba5afb643f5302e7ff751ffe12d85e4a9b3db35a703055bef"} err="failed to get container status \"a6b091b86021cc4ba5afb643f5302e7ff751ffe12d85e4a9b3db35a703055bef\": rpc error: code = NotFound desc = could not find container \"a6b091b86021cc4ba5afb643f5302e7ff751ffe12d85e4a9b3db35a703055bef\": container with ID starting with a6b091b86021cc4ba5afb643f5302e7ff751ffe12d85e4a9b3db35a703055bef not found: ID does not exist"
Sep 09 16:36:25.265632 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.265606    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"]
Sep 09 16:36:25.268574 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.268552    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-042f8b-cf6b9c498-8rvq6"]
Sep 09 16:36:25.278090 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.278063    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"]
Sep 09 16:36:25.283414 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:25.283396    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-042f8b-658d849bc7-w8thl"]
Sep 09 16:36:26.553038 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:26.552999    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fbafabb-8232-41f8-bbaf-f0c4b0e137c6" path="/var/lib/kubelet/pods/4fbafabb-8232-41f8-bbaf-f0c4b0e137c6/volumes"
Sep 09 16:36:26.553445 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:26.553338    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4dd7c8c-f218-4f30-b6b6-087f4a1fa106" path="/var/lib/kubelet/pods/b4dd7c8c-f218-4f30-b6b6-087f4a1fa106/volumes"
Sep 09 16:36:26.553627 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:26.553615    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb09db09-f69c-4dad-a7dc-4ce7fe732674" path="/var/lib/kubelet/pods/eb09db09-f69c-4dad-a7dc-4ce7fe732674/volumes"
Sep 09 16:36:31.601671 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.601636    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67"]
Sep 09 16:36:31.602068 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.601981    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb09db09-f69c-4dad-a7dc-4ce7fe732674" containerName="payload-processing"
Sep 09 16:36:31.602068 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.601993    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb09db09-f69c-4dad-a7dc-4ce7fe732674" containerName="payload-processing"
Sep 09 16:36:31.602068 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.602006    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fbafabb-8232-41f8-bbaf-f0c4b0e137c6" containerName="revoke-keys"
Sep 09 16:36:31.602068 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.602012    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fbafabb-8232-41f8-bbaf-f0c4b0e137c6" containerName="revoke-keys"
Sep 09 16:36:31.602068 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.602022    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4dd7c8c-f218-4f30-b6b6-087f4a1fa106" containerName="payload-pre-processing"
Sep 09 16:36:31.602068 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.602031    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4dd7c8c-f218-4f30-b6b6-087f4a1fa106" containerName="payload-pre-processing"
Sep 09 16:36:31.602258 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.602080    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fbafabb-8232-41f8-bbaf-f0c4b0e137c6" containerName="revoke-keys"
Sep 09 16:36:31.602258 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.602090    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb09db09-f69c-4dad-a7dc-4ce7fe732674" containerName="payload-processing"
Sep 09 16:36:31.602258 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.602097    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4dd7c8c-f218-4f30-b6b6-087f4a1fa106" containerName="payload-pre-processing"
Sep 09 16:36:31.606216 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.606200    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67"
Sep 09 16:36:31.608588 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.608569    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:36:31.608772 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.608757    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxghs\""
Sep 09 16:36:31.612222 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.612191    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67"]
Sep 09 16:36:31.665799 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.665759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e688a9d-a127-45eb-b328-be4e3d61d83e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67\" (UID: \"4e688a9d-a127-45eb-b328-be4e3d61d83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67"
Sep 09 16:36:31.766214 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.766179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e688a9d-a127-45eb-b328-be4e3d61d83e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67\" (UID: \"4e688a9d-a127-45eb-b328-be4e3d61d83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67"
Sep 09 16:36:31.766815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.766796    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e688a9d-a127-45eb-b328-be4e3d61d83e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67\" (UID: \"4e688a9d-a127-45eb-b328-be4e3d61d83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67"
Sep 09 16:36:31.917980 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:31.917899    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67"
Sep 09 16:36:32.187399 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:32.187327    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67"]
Sep 09 16:36:32.190721 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:36:32.190676    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e688a9d_a127_45eb_b328_be4e3d61d83e.slice/crio-db78b434d9eb87c728c73039ce7dc7e515f9f0f3fb9e70d3c7d7002831e9a424 WatchSource:0}: Error finding container db78b434d9eb87c728c73039ce7dc7e515f9f0f3fb9e70d3c7d7002831e9a424: Status 404 returned error can't find the container with id db78b434d9eb87c728c73039ce7dc7e515f9f0f3fb9e70d3c7d7002831e9a424
Sep 09 16:36:32.277761 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:32.277723    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67" event={"ID":"4e688a9d-a127-45eb-b328-be4e3d61d83e","Type":"ContainerStarted","Data":"db78b434d9eb87c728c73039ce7dc7e515f9f0f3fb9e70d3c7d7002831e9a424"}
Sep 09 16:36:33.284017 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:33.283986    2576 generic.go:356] "Generic (PLEG): container finished" podID="4e688a9d-a127-45eb-b328-be4e3d61d83e" containerID="28453f323bc093b7bbef7963716d5457bc126ebbf6c93ae549a84165fb4cb1a4" exitCode=0
Sep 09 16:36:33.284399 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:33.284076    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67" event={"ID":"4e688a9d-a127-45eb-b328-be4e3d61d83e","Type":"ContainerDied","Data":"28453f323bc093b7bbef7963716d5457bc126ebbf6c93ae549a84165fb4cb1a4"}
Sep 09 16:36:34.416472 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:34.416449    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67"
Sep 09 16:36:34.484957 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:34.484921    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e688a9d-a127-45eb-b328-be4e3d61d83e-maas-api-service-ca\") pod \"4e688a9d-a127-45eb-b328-be4e3d61d83e\" (UID: \"4e688a9d-a127-45eb-b328-be4e3d61d83e\") "
Sep 09 16:36:34.485273 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:34.485251    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e688a9d-a127-45eb-b328-be4e3d61d83e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e688a9d-a127-45eb-b328-be4e3d61d83e" (UID: "4e688a9d-a127-45eb-b328-be4e3d61d83e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:36:34.585569 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:34.585534    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e688a9d-a127-45eb-b328-be4e3d61d83e-maas-api-service-ca\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:36:35.297645 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:35.297617    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67"
Sep 09 16:36:35.297857 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:35.297651    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67" event={"ID":"4e688a9d-a127-45eb-b328-be4e3d61d83e","Type":"ContainerDied","Data":"db78b434d9eb87c728c73039ce7dc7e515f9f0f3fb9e70d3c7d7002831e9a424"}
Sep 09 16:36:35.297857 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:35.297727    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db78b434d9eb87c728c73039ce7dc7e515f9f0f3fb9e70d3c7d7002831e9a424"
Sep 09 16:36:42.266949 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:42.266911    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67"]
Sep 09 16:36:42.269630 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:42.269601    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de97ae-mtz67"]
Sep 09 16:36:42.553177 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:42.553098    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e688a9d-a127-45eb-b328-be4e3d61d83e" path="/var/lib/kubelet/pods/4e688a9d-a127-45eb-b328-be4e3d61d83e/volumes"
Sep 09 16:36:44.728036 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:36:44.728004    2576 scope.go:117] "RemoveContainer" containerID="830cfac61b0733bb6edc3a07102b06d4b6742968e3a30956f91763600f53ba9c"
Sep 09 16:37:02.651308 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:02.651275    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"]
Sep 09 16:37:03.733267 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:03.733236    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"]
Sep 09 16:37:17.671264 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:17.671162    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p" podUID="bdbc0851-7ea9-41e3-b8db-c39c446300da" containerName="main" containerID="cri-o://cefebdc9927040430cd03f2bf2989fc16678f395e1cbf46ff49ac5f34d08fb38" gracePeriod=45
Sep 09 16:37:17.921808 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:17.921743    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:37:17.973689 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:17.973664    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-dshm\") pod \"bdbc0851-7ea9-41e3-b8db-c39c446300da\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") "
Sep 09 16:37:17.973896 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:17.973723    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdbc0851-7ea9-41e3-b8db-c39c446300da-tls-certs\") pod \"bdbc0851-7ea9-41e3-b8db-c39c446300da\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") "
Sep 09 16:37:17.973896 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:17.973764    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wrg4\" (UniqueName: \"kubernetes.io/projected/bdbc0851-7ea9-41e3-b8db-c39c446300da-kube-api-access-5wrg4\") pod \"bdbc0851-7ea9-41e3-b8db-c39c446300da\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") "
Sep 09 16:37:17.973896 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:17.973801    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-model-cache\") pod \"bdbc0851-7ea9-41e3-b8db-c39c446300da\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") "
Sep 09 16:37:17.973896 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:17.973825    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-home\") pod \"bdbc0851-7ea9-41e3-b8db-c39c446300da\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") "
Sep 09 16:37:17.973896 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:17.973851    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-tmp-dir\") pod \"bdbc0851-7ea9-41e3-b8db-c39c446300da\" (UID: \"bdbc0851-7ea9-41e3-b8db-c39c446300da\") "
Sep 09 16:37:17.974179 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:17.974141    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-home" (OuterVolumeSpecName: "home") pod "bdbc0851-7ea9-41e3-b8db-c39c446300da" (UID: "bdbc0851-7ea9-41e3-b8db-c39c446300da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:17.974275 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:17.974243    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdbc0851-7ea9-41e3-b8db-c39c446300da" (UID: "bdbc0851-7ea9-41e3-b8db-c39c446300da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:17.974589 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:17.974568    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdbc0851-7ea9-41e3-b8db-c39c446300da" (UID: "bdbc0851-7ea9-41e3-b8db-c39c446300da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:17.976273 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:17.976256    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-dshm" (OuterVolumeSpecName: "dshm") pod "bdbc0851-7ea9-41e3-b8db-c39c446300da" (UID: "bdbc0851-7ea9-41e3-b8db-c39c446300da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:17.976470 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:17.976438    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdbc0851-7ea9-41e3-b8db-c39c446300da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdbc0851-7ea9-41e3-b8db-c39c446300da" (UID: "bdbc0851-7ea9-41e3-b8db-c39c446300da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:37:17.976470 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:17.976448    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbc0851-7ea9-41e3-b8db-c39c446300da-kube-api-access-5wrg4" (OuterVolumeSpecName: "kube-api-access-5wrg4") pod "bdbc0851-7ea9-41e3-b8db-c39c446300da" (UID: "bdbc0851-7ea9-41e3-b8db-c39c446300da"). InnerVolumeSpecName "kube-api-access-5wrg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:18.075077 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.075032    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdbc0851-7ea9-41e3-b8db-c39c446300da-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:37:18.075077 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.075071    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wrg4\" (UniqueName: \"kubernetes.io/projected/bdbc0851-7ea9-41e3-b8db-c39c446300da-kube-api-access-5wrg4\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:37:18.075077 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.075083    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:37:18.075077 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.075092    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:37:18.075333 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.075100    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:37:18.075333 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.075107    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdbc0851-7ea9-41e3-b8db-c39c446300da-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:37:18.497563 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.497524    2576 generic.go:356] "Generic (PLEG): container finished" podID="bdbc0851-7ea9-41e3-b8db-c39c446300da" containerID="cefebdc9927040430cd03f2bf2989fc16678f395e1cbf46ff49ac5f34d08fb38" exitCode=0
Sep 09 16:37:18.497806 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.497588    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"
Sep 09 16:37:18.497806 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.497616    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p" event={"ID":"bdbc0851-7ea9-41e3-b8db-c39c446300da","Type":"ContainerDied","Data":"cefebdc9927040430cd03f2bf2989fc16678f395e1cbf46ff49ac5f34d08fb38"}
Sep 09 16:37:18.497806 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.497665    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p" event={"ID":"bdbc0851-7ea9-41e3-b8db-c39c446300da","Type":"ContainerDied","Data":"8bc8d1f5b36efafa34d448e6c6a0d1c745a352c407157dd245d07231439a5d54"}
Sep 09 16:37:18.497806 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.497688    2576 scope.go:117] "RemoveContainer" containerID="cefebdc9927040430cd03f2bf2989fc16678f395e1cbf46ff49ac5f34d08fb38"
Sep 09 16:37:18.506541 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.506522    2576 scope.go:117] "RemoveContainer" containerID="cefebdc9927040430cd03f2bf2989fc16678f395e1cbf46ff49ac5f34d08fb38"
Sep 09 16:37:18.506852 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:37:18.506833    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefebdc9927040430cd03f2bf2989fc16678f395e1cbf46ff49ac5f34d08fb38\": container with ID starting with cefebdc9927040430cd03f2bf2989fc16678f395e1cbf46ff49ac5f34d08fb38 not found: ID does not exist" containerID="cefebdc9927040430cd03f2bf2989fc16678f395e1cbf46ff49ac5f34d08fb38"
Sep 09 16:37:18.506908 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.506861    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefebdc9927040430cd03f2bf2989fc16678f395e1cbf46ff49ac5f34d08fb38"} err="failed to get container status \"cefebdc9927040430cd03f2bf2989fc16678f395e1cbf46ff49ac5f34d08fb38\": rpc error: code = NotFound desc = could not find container \"cefebdc9927040430cd03f2bf2989fc16678f395e1cbf46ff49ac5f34d08fb38\": container with ID starting with cefebdc9927040430cd03f2bf2989fc16678f395e1cbf46ff49ac5f34d08fb38 not found: ID does not exist"
Sep 09 16:37:18.517356 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.517335    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"]
Sep 09 16:37:18.521708 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.521676    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-041bf1/rate-test-model-041bf1-kserve-7978954c79-vl54p"]
Sep 09 16:37:18.552941 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.552916    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdbc0851-7ea9-41e3-b8db-c39c446300da" path="/var/lib/kubelet/pods/bdbc0851-7ea9-41e3-b8db-c39c446300da/volumes"
Sep 09 16:37:18.752839 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.752754    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9" podUID="9e2db2a1-72ff-4750-bab2-946632df5c71" containerName="main" containerID="cri-o://dff77c0f380048bc5c63f4aeb98216a6cc0de963e55fdfcfa0775567ca057f94" gracePeriod=45
Sep 09 16:37:18.998622 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:18.998594    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:37:19.084022 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.083930    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4dwz\" (UniqueName: \"kubernetes.io/projected/9e2db2a1-72ff-4750-bab2-946632df5c71-kube-api-access-p4dwz\") pod \"9e2db2a1-72ff-4750-bab2-946632df5c71\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") "
Sep 09 16:37:19.084022 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.083975    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-home\") pod \"9e2db2a1-72ff-4750-bab2-946632df5c71\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") "
Sep 09 16:37:19.084022 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.084007    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-model-cache\") pod \"9e2db2a1-72ff-4750-bab2-946632df5c71\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") "
Sep 09 16:37:19.084298 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.084043    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-dshm\") pod \"9e2db2a1-72ff-4750-bab2-946632df5c71\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") "
Sep 09 16:37:19.084298 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.084090    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-tmp-dir\") pod \"9e2db2a1-72ff-4750-bab2-946632df5c71\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") "
Sep 09 16:37:19.084298 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.084128    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2db2a1-72ff-4750-bab2-946632df5c71-tls-certs\") pod \"9e2db2a1-72ff-4750-bab2-946632df5c71\" (UID: \"9e2db2a1-72ff-4750-bab2-946632df5c71\") "
Sep 09 16:37:19.084298 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.084252    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-home" (OuterVolumeSpecName: "home") pod "9e2db2a1-72ff-4750-bab2-946632df5c71" (UID: "9e2db2a1-72ff-4750-bab2-946632df5c71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:19.084469 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.084297    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e2db2a1-72ff-4750-bab2-946632df5c71" (UID: "9e2db2a1-72ff-4750-bab2-946632df5c71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:19.084469 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.084373    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e2db2a1-72ff-4750-bab2-946632df5c71" (UID: "9e2db2a1-72ff-4750-bab2-946632df5c71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:19.084469 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.084445    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:37:19.084469 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.084465    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:37:19.086274 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.086253    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-dshm" (OuterVolumeSpecName: "dshm") pod "9e2db2a1-72ff-4750-bab2-946632df5c71" (UID: "9e2db2a1-72ff-4750-bab2-946632df5c71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:19.086360 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.086296    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e2db2a1-72ff-4750-bab2-946632df5c71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e2db2a1-72ff-4750-bab2-946632df5c71" (UID: "9e2db2a1-72ff-4750-bab2-946632df5c71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:37:19.086398 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.086358    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2db2a1-72ff-4750-bab2-946632df5c71-kube-api-access-p4dwz" (OuterVolumeSpecName: "kube-api-access-p4dwz") pod "9e2db2a1-72ff-4750-bab2-946632df5c71" (UID: "9e2db2a1-72ff-4750-bab2-946632df5c71"). InnerVolumeSpecName "kube-api-access-p4dwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:19.185620 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.185587    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:37:19.185620 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.185613    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2db2a1-72ff-4750-bab2-946632df5c71-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:37:19.185620 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.185623    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4dwz\" (UniqueName: \"kubernetes.io/projected/9e2db2a1-72ff-4750-bab2-946632df5c71-kube-api-access-p4dwz\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:37:19.185886 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.185631    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2db2a1-72ff-4750-bab2-946632df5c71-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:37:19.504637 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.504601    2576 generic.go:356] "Generic (PLEG): container finished" podID="9e2db2a1-72ff-4750-bab2-946632df5c71" containerID="dff77c0f380048bc5c63f4aeb98216a6cc0de963e55fdfcfa0775567ca057f94" exitCode=0
Sep 09 16:37:19.504838 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.504644    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"
Sep 09 16:37:19.504838 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.504643    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9" event={"ID":"9e2db2a1-72ff-4750-bab2-946632df5c71","Type":"ContainerDied","Data":"dff77c0f380048bc5c63f4aeb98216a6cc0de963e55fdfcfa0775567ca057f94"}
Sep 09 16:37:19.504838 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.504768    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9" event={"ID":"9e2db2a1-72ff-4750-bab2-946632df5c71","Type":"ContainerDied","Data":"a42128d23b60eac30c8785581a8ac076b544cb317a4975c7f0a3baeedcc4d4c9"}
Sep 09 16:37:19.504838 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.504792    2576 scope.go:117] "RemoveContainer" containerID="dff77c0f380048bc5c63f4aeb98216a6cc0de963e55fdfcfa0775567ca057f94"
Sep 09 16:37:19.513738 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.513718    2576 scope.go:117] "RemoveContainer" containerID="dff77c0f380048bc5c63f4aeb98216a6cc0de963e55fdfcfa0775567ca057f94"
Sep 09 16:37:19.514039 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:37:19.514017    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff77c0f380048bc5c63f4aeb98216a6cc0de963e55fdfcfa0775567ca057f94\": container with ID starting with dff77c0f380048bc5c63f4aeb98216a6cc0de963e55fdfcfa0775567ca057f94 not found: ID does not exist" containerID="dff77c0f380048bc5c63f4aeb98216a6cc0de963e55fdfcfa0775567ca057f94"
Sep 09 16:37:19.514089 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.514049    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff77c0f380048bc5c63f4aeb98216a6cc0de963e55fdfcfa0775567ca057f94"} err="failed to get container status \"dff77c0f380048bc5c63f4aeb98216a6cc0de963e55fdfcfa0775567ca057f94\": rpc error: code = NotFound desc = could not find container \"dff77c0f380048bc5c63f4aeb98216a6cc0de963e55fdfcfa0775567ca057f94\": container with ID starting with dff77c0f380048bc5c63f4aeb98216a6cc0de963e55fdfcfa0775567ca057f94 not found: ID does not exist"
Sep 09 16:37:19.524400 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.524376    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"]
Sep 09 16:37:19.530082 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:19.530060    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9520a4/rate-test-model-9520a4-kserve-6b4fd94599-vf2t9"]
Sep 09 16:37:20.553635 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:20.553600    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e2db2a1-72ff-4750-bab2-946632df5c71" path="/var/lib/kubelet/pods/9e2db2a1-72ff-4750-bab2-946632df5c71/volumes"
Sep 09 16:37:38.952770 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.952726    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"]
Sep 09 16:37:38.953235 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.953051    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e2db2a1-72ff-4750-bab2-946632df5c71" containerName="main"
Sep 09 16:37:38.953235 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.953062    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e2db2a1-72ff-4750-bab2-946632df5c71" containerName="main"
Sep 09 16:37:38.953235 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.953070    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e688a9d-a127-45eb-b328-be4e3d61d83e" containerName="revoke-keys"
Sep 09 16:37:38.953235 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.953075    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e688a9d-a127-45eb-b328-be4e3d61d83e" containerName="revoke-keys"
Sep 09 16:37:38.953235 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.953094    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdbc0851-7ea9-41e3-b8db-c39c446300da" containerName="main"
Sep 09 16:37:38.953235 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.953100    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdbc0851-7ea9-41e3-b8db-c39c446300da" containerName="main"
Sep 09 16:37:38.953235 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.953146    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdbc0851-7ea9-41e3-b8db-c39c446300da" containerName="main"
Sep 09 16:37:38.953235 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.953155    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e2db2a1-72ff-4750-bab2-946632df5c71" containerName="main"
Sep 09 16:37:38.953235 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.953162    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e688a9d-a127-45eb-b328-be4e3d61d83e" containerName="revoke-keys"
Sep 09 16:37:38.957358 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.957339    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"
Sep 09 16:37:38.959520 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.959498    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3c6036\""
Sep 09 16:37:38.959742 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.959728    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3c6036-dockercfg-vfzt2\""
Sep 09 16:37:38.966652 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.966631    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"]
Sep 09 16:37:38.986350 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.986319    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"]
Sep 09 16:37:38.989605 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.989587    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"
Sep 09 16:37:38.998635 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:38.998611    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"]
Sep 09 16:37:39.053312 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.053277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf403f4-0c5d-4c72-b002-b4fbee774bc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q\" (UID: \"9bf403f4-0c5d-4c72-b002-b4fbee774bc3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"
Sep 09 16:37:39.053312 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.053315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2swr2\" (UniqueName: \"kubernetes.io/projected/9bf403f4-0c5d-4c72-b002-b4fbee774bc3-kube-api-access-2swr2\") pod \"payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q\" (UID: \"9bf403f4-0c5d-4c72-b002-b4fbee774bc3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"
Sep 09 16:37:39.053557 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.053395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33c68b8-749c-4a69-96b6-cdc447af2192-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw\" (UID: \"e33c68b8-749c-4a69-96b6-cdc447af2192\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"
Sep 09 16:37:39.053557 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.053491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dhzk\" (UniqueName: \"kubernetes.io/projected/e33c68b8-749c-4a69-96b6-cdc447af2192-kube-api-access-8dhzk\") pod \"payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw\" (UID: \"e33c68b8-749c-4a69-96b6-cdc447af2192\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"
Sep 09 16:37:39.154892 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.154852    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33c68b8-749c-4a69-96b6-cdc447af2192-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw\" (UID: \"e33c68b8-749c-4a69-96b6-cdc447af2192\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"
Sep 09 16:37:39.155082 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.154907    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dhzk\" (UniqueName: \"kubernetes.io/projected/e33c68b8-749c-4a69-96b6-cdc447af2192-kube-api-access-8dhzk\") pod \"payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw\" (UID: \"e33c68b8-749c-4a69-96b6-cdc447af2192\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"
Sep 09 16:37:39.155082 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.154933    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf403f4-0c5d-4c72-b002-b4fbee774bc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q\" (UID: \"9bf403f4-0c5d-4c72-b002-b4fbee774bc3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"
Sep 09 16:37:39.155082 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.154957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2swr2\" (UniqueName: \"kubernetes.io/projected/9bf403f4-0c5d-4c72-b002-b4fbee774bc3-kube-api-access-2swr2\") pod \"payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q\" (UID: \"9bf403f4-0c5d-4c72-b002-b4fbee774bc3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"
Sep 09 16:37:39.155801 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.155776    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33c68b8-749c-4a69-96b6-cdc447af2192-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw\" (UID: \"e33c68b8-749c-4a69-96b6-cdc447af2192\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"
Sep 09 16:37:39.155863 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.155776    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf403f4-0c5d-4c72-b002-b4fbee774bc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q\" (UID: \"9bf403f4-0c5d-4c72-b002-b4fbee774bc3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"
Sep 09 16:37:39.162977 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.162953    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dhzk\" (UniqueName: \"kubernetes.io/projected/e33c68b8-749c-4a69-96b6-cdc447af2192-kube-api-access-8dhzk\") pod \"payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw\" (UID: \"e33c68b8-749c-4a69-96b6-cdc447af2192\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"
Sep 09 16:37:39.163084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.163032    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2swr2\" (UniqueName: \"kubernetes.io/projected/9bf403f4-0c5d-4c72-b002-b4fbee774bc3-kube-api-access-2swr2\") pod \"payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q\" (UID: \"9bf403f4-0c5d-4c72-b002-b4fbee774bc3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"
Sep 09 16:37:39.269137 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.269102    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"
Sep 09 16:37:39.299492 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.299460    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"
Sep 09 16:37:39.543806 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.543777    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"]
Sep 09 16:37:39.545709 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:37:39.545671    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf403f4_0c5d_4c72_b002_b4fbee774bc3.slice/crio-4dad803e89cb6cd2b423376c21617a23d23b96cb0f7ed7776cc71264e9a38800 WatchSource:0}: Error finding container 4dad803e89cb6cd2b423376c21617a23d23b96cb0f7ed7776cc71264e9a38800: Status 404 returned error can't find the container with id 4dad803e89cb6cd2b423376c21617a23d23b96cb0f7ed7776cc71264e9a38800
Sep 09 16:37:39.583968 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.583907    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"]
Sep 09 16:37:39.586064 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:37:39.586025    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33c68b8_749c_4a69_96b6_cdc447af2192.slice/crio-b4b81c4841b9ed522524ebaee19f438bdaa18730e6cda845bf9bb1746d8ceb29 WatchSource:0}: Error finding container b4b81c4841b9ed522524ebaee19f438bdaa18730e6cda845bf9bb1746d8ceb29: Status 404 returned error can't find the container with id b4b81c4841b9ed522524ebaee19f438bdaa18730e6cda845bf9bb1746d8ceb29
Sep 09 16:37:39.600769 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.600739    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw" event={"ID":"e33c68b8-749c-4a69-96b6-cdc447af2192","Type":"ContainerStarted","Data":"b4b81c4841b9ed522524ebaee19f438bdaa18730e6cda845bf9bb1746d8ceb29"}
Sep 09 16:37:39.602167 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:39.602144    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q" event={"ID":"9bf403f4-0c5d-4c72-b002-b4fbee774bc3","Type":"ContainerStarted","Data":"4dad803e89cb6cd2b423376c21617a23d23b96cb0f7ed7776cc71264e9a38800"}
Sep 09 16:37:40.607966 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:40.607924    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw" event={"ID":"e33c68b8-749c-4a69-96b6-cdc447af2192","Type":"ContainerStarted","Data":"2cb60b46acaa4692f9da93918192cb379e441de2a0df7c1cef0488e8294ca5c7"}
Sep 09 16:37:40.608412 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:40.607986    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"
Sep 09 16:37:40.609578 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:40.609553    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q" event={"ID":"9bf403f4-0c5d-4c72-b002-b4fbee774bc3","Type":"ContainerStarted","Data":"ce72aa77a2347122175e1f050fb003af28e19c1bc162e6b12dfae70d1a64cf81"}
Sep 09 16:37:40.609744 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:40.609668    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"
Sep 09 16:37:40.623823 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:40.623781    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw" podStartSLOduration=2.62376807 podStartE2EDuration="2.62376807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:37:40.62266887 +0000 UTC m=+1736.558287285" watchObservedRunningTime="2026-09-09 16:37:40.62376807 +0000 UTC m=+1736.559386538"
Sep 09 16:37:40.637634 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:40.637590    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q" podStartSLOduration=2.637576542 podStartE2EDuration="2.637576542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:37:40.636430619 +0000 UTC m=+1736.572049053" watchObservedRunningTime="2026-09-09 16:37:40.637576542 +0000 UTC m=+1736.573194959"
Sep 09 16:37:44.760978 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:44.760948    2576 scope.go:117] "RemoveContainer" containerID="924af2be4c68138f9d3fe8a281f2a4ce995451f4e1474c1b2fcd1be22b707c5d"
Sep 09 16:37:51.616747 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:51.616689    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"
Sep 09 16:37:51.617187 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:37:51.616797    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"
Sep 09 16:38:07.598265 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:07.598235    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-s76z5_6bcd5da5-f6c4-46aa-87dd-21844af21310/payload-processing/0.log"
Sep 09 16:38:07.747245 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:07.747217    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw_e33c68b8-749c-4a69-96b6-cdc447af2192/payload-processing/0.log"
Sep 09 16:38:13.817730 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.817665    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"]
Sep 09 16:38:13.821169 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.821147    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.825079 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.825044    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-54370d\"/\"kube-root-ca.crt\""
Sep 09 16:38:13.825079 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.825059    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-54370d\"/\"openshift-service-ca.crt\""
Sep 09 16:38:13.825264 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.825078    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-54370d\"/\"ipp-route-54370d-kserve-self-signed-certs\""
Sep 09 16:38:13.825264 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.825044    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-54370d\"/\"default-dockercfg-4sdhc\""
Sep 09 16:38:13.828958 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.828936    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"]
Sep 09 16:38:13.860815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.860779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-dshm\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.860815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.860813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-tmp-dir\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.860976 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.860845    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhfk2\" (UniqueName: \"kubernetes.io/projected/77abe4c7-6a16-41b6-b0c9-c28535beae96-kube-api-access-jhfk2\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.860976 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.860931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-home\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.860976 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.860959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77abe4c7-6a16-41b6-b0c9-c28535beae96-tls-certs\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.861091 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.860986    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-model-cache\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.961684 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.961646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-dshm\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.961684 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.961683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-tmp-dir\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.961958 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.961749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhfk2\" (UniqueName: \"kubernetes.io/projected/77abe4c7-6a16-41b6-b0c9-c28535beae96-kube-api-access-jhfk2\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.961958 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.961814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-home\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.961958 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.961839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77abe4c7-6a16-41b6-b0c9-c28535beae96-tls-certs\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.961958 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.961872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-model-cache\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.962159 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.962139    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-tmp-dir\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.962212 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.962180    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-home\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.962212 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.962199    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-model-cache\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.964015 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.963996    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-dshm\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.964319 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.964300    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77abe4c7-6a16-41b6-b0c9-c28535beae96-tls-certs\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:13.969461 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:13.969438    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhfk2\" (UniqueName: \"kubernetes.io/projected/77abe4c7-6a16-41b6-b0c9-c28535beae96-kube-api-access-jhfk2\") pod \"ipp-route-54370d-kserve-d6bbd9767-2b7wd\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") " pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:14.133320 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:14.133231    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:14.404488 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:14.404465    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"]
Sep 09 16:38:14.406834 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:38:14.406807    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77abe4c7_6a16_41b6_b0c9_c28535beae96.slice/crio-77b57225d58dbaa65ab6a4e14cac61a6f69631d3d844d6995246e56fbf061e61 WatchSource:0}: Error finding container 77b57225d58dbaa65ab6a4e14cac61a6f69631d3d844d6995246e56fbf061e61: Status 404 returned error can't find the container with id 77b57225d58dbaa65ab6a4e14cac61a6f69631d3d844d6995246e56fbf061e61
Sep 09 16:38:14.768031 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:14.767996    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd" event={"ID":"77abe4c7-6a16-41b6-b0c9-c28535beae96","Type":"ContainerStarted","Data":"24df2430b20c1aec9bf8605770e6ef38b1658f0b63d25310c6ef1a0a08213456"}
Sep 09 16:38:14.768031 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:14.768036    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd" event={"ID":"77abe4c7-6a16-41b6-b0c9-c28535beae96","Type":"ContainerStarted","Data":"77b57225d58dbaa65ab6a4e14cac61a6f69631d3d844d6995246e56fbf061e61"}
Sep 09 16:38:14.784299 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:14.784246    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd" podStartSLOduration=1.784231617 podStartE2EDuration="1.784231617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:38:14.78290243 +0000 UTC m=+1770.718520865" watchObservedRunningTime="2026-09-09 16:38:14.784231617 +0000 UTC m=+1770.719850032"
Sep 09 16:38:24.133789 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:24.133747    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:24.134262 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:24.133807    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:24.145927 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:24.145905    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:24.825665 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:24.825631    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:38:40.787172 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:40.787087    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-s76z5_6bcd5da5-f6c4-46aa-87dd-21844af21310/payload-processing/0.log"
Sep 09 16:38:40.912552 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:40.912528    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw_e33c68b8-749c-4a69-96b6-cdc447af2192/payload-processing/0.log"
Sep 09 16:38:44.784894 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:44.784858    2576 scope.go:117] "RemoveContainer" containerID="dc12d03e3cd8b8683d5cc924a15f2dd89d5c598b43da1010b45a15add2a35685"
Sep 09 16:38:46.502215 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.502181    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"]
Sep 09 16:38:46.505604 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.505586    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"
Sep 09 16:38:46.512126 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.512069    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-936acc\""
Sep 09 16:38:46.520011 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.519987    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"]
Sep 09 16:38:46.545504 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.545471    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eb455ef-0b5b-42f5-8e01-dc84246dd8e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt\" (UID: \"7eb455ef-0b5b-42f5-8e01-dc84246dd8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"
Sep 09 16:38:46.545650 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.545509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v857\" (UniqueName: \"kubernetes.io/projected/7eb455ef-0b5b-42f5-8e01-dc84246dd8e0-kube-api-access-2v857\") pod \"payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt\" (UID: \"7eb455ef-0b5b-42f5-8e01-dc84246dd8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"
Sep 09 16:38:46.556414 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.556389    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"]
Sep 09 16:38:46.560207 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.560184    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"
Sep 09 16:38:46.562719 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.562678    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-936acc-dockercfg-4wdzw\""
Sep 09 16:38:46.564254 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.564232    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"]
Sep 09 16:38:46.646465 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.646424    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgqbj\" (UniqueName: \"kubernetes.io/projected/465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f-kube-api-access-dgqbj\") pod \"payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6\" (UID: \"465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"
Sep 09 16:38:46.646653 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.646510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eb455ef-0b5b-42f5-8e01-dc84246dd8e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt\" (UID: \"7eb455ef-0b5b-42f5-8e01-dc84246dd8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"
Sep 09 16:38:46.646653 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.646543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v857\" (UniqueName: \"kubernetes.io/projected/7eb455ef-0b5b-42f5-8e01-dc84246dd8e0-kube-api-access-2v857\") pod \"payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt\" (UID: \"7eb455ef-0b5b-42f5-8e01-dc84246dd8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"
Sep 09 16:38:46.646653 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.646564    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6\" (UID: \"465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"
Sep 09 16:38:46.647183 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.647165    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eb455ef-0b5b-42f5-8e01-dc84246dd8e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt\" (UID: \"7eb455ef-0b5b-42f5-8e01-dc84246dd8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"
Sep 09 16:38:46.654613 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.654591    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v857\" (UniqueName: \"kubernetes.io/projected/7eb455ef-0b5b-42f5-8e01-dc84246dd8e0-kube-api-access-2v857\") pod \"payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt\" (UID: \"7eb455ef-0b5b-42f5-8e01-dc84246dd8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"
Sep 09 16:38:46.747904 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.747861    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6\" (UID: \"465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"
Sep 09 16:38:46.748088 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.747953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgqbj\" (UniqueName: \"kubernetes.io/projected/465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f-kube-api-access-dgqbj\") pod \"payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6\" (UID: \"465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"
Sep 09 16:38:46.748597 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.748577    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6\" (UID: \"465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"
Sep 09 16:38:46.756682 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.756622    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgqbj\" (UniqueName: \"kubernetes.io/projected/465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f-kube-api-access-dgqbj\") pod \"payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6\" (UID: \"465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"
Sep 09 16:38:46.815466 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.815432    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"
Sep 09 16:38:46.870618 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:46.870585    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"
Sep 09 16:38:47.087326 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:47.087300    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"]
Sep 09 16:38:47.089356 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:38:47.089328    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb455ef_0b5b_42f5_8e01_dc84246dd8e0.slice/crio-164ebc29be47a5b50441fc0c01340088d0caf42f5be21bf24f80c5afe1984b35 WatchSource:0}: Error finding container 164ebc29be47a5b50441fc0c01340088d0caf42f5be21bf24f80c5afe1984b35: Status 404 returned error can't find the container with id 164ebc29be47a5b50441fc0c01340088d0caf42f5be21bf24f80c5afe1984b35
Sep 09 16:38:47.148653 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:47.148623    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"]
Sep 09 16:38:47.151482 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:38:47.151456    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465bd2a2_fbb2_45eb_9d7a_49a5e0f75c2f.slice/crio-b2b3c63b838c990f3a2e5b637b664a5d97285098577f9be1e8704375c1ff171b WatchSource:0}: Error finding container b2b3c63b838c990f3a2e5b637b664a5d97285098577f9be1e8704375c1ff171b: Status 404 returned error can't find the container with id b2b3c63b838c990f3a2e5b637b664a5d97285098577f9be1e8704375c1ff171b
Sep 09 16:38:47.919186 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:47.919147    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt" event={"ID":"7eb455ef-0b5b-42f5-8e01-dc84246dd8e0","Type":"ContainerStarted","Data":"ad33524bdedf00457ccd78d4ce61ec9cc218af8a9581aa77a52d45627ea18637"}
Sep 09 16:38:47.919597 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:47.919215    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt" event={"ID":"7eb455ef-0b5b-42f5-8e01-dc84246dd8e0","Type":"ContainerStarted","Data":"164ebc29be47a5b50441fc0c01340088d0caf42f5be21bf24f80c5afe1984b35"}
Sep 09 16:38:47.919597 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:47.919244    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"
Sep 09 16:38:47.920774 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:47.920752    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6" event={"ID":"465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f","Type":"ContainerStarted","Data":"efcd3bb5c8e15a9747ae3ee11956b3516528aaf0054f418bc1fca5314b7e0770"}
Sep 09 16:38:47.920889 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:47.920790    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6" event={"ID":"465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f","Type":"ContainerStarted","Data":"b2b3c63b838c990f3a2e5b637b664a5d97285098577f9be1e8704375c1ff171b"}
Sep 09 16:38:47.920889 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:47.920812    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"
Sep 09 16:38:47.937897 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:47.937845    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt" podStartSLOduration=1.937828656 podStartE2EDuration="1.937828656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:38:47.935930403 +0000 UTC m=+1803.871548859" watchObservedRunningTime="2026-09-09 16:38:47.937828656 +0000 UTC m=+1803.873447073"
Sep 09 16:38:47.951656 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:47.951609    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6" podStartSLOduration=1.951593948 podStartE2EDuration="1.951593948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:38:47.950416288 +0000 UTC m=+1803.886034724" watchObservedRunningTime="2026-09-09 16:38:47.951593948 +0000 UTC m=+1803.887212364"
Sep 09 16:38:58.927370 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:58.927331    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"
Sep 09 16:38:58.927880 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:38:58.927579    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"
Sep 09 16:39:10.574420 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:10.574386    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68"]
Sep 09 16:39:10.577929 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:10.577912    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68"
Sep 09 16:39:10.580519 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:10.580495    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:39:10.580519 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:10.580509    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxghs\""
Sep 09 16:39:10.584640 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:10.584619    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68"]
Sep 09 16:39:10.650838 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:10.650810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11313a0b-f798-4005-954b-fd2628ff668f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68\" (UID: \"11313a0b-f798-4005-954b-fd2628ff668f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68"
Sep 09 16:39:10.752106 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:10.752074    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11313a0b-f798-4005-954b-fd2628ff668f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68\" (UID: \"11313a0b-f798-4005-954b-fd2628ff668f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68"
Sep 09 16:39:10.752771 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:10.752751    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11313a0b-f798-4005-954b-fd2628ff668f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68\" (UID: \"11313a0b-f798-4005-954b-fd2628ff668f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68"
Sep 09 16:39:10.889877 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:10.889795    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68"
Sep 09 16:39:11.160460 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:11.160436    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68"]
Sep 09 16:39:11.162484 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:39:11.162457    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11313a0b_f798_4005_954b_fd2628ff668f.slice/crio-3f520234cf9b24a306829ecd80953cca87b6d6280f19e011cc8c49a784507a7f WatchSource:0}: Error finding container 3f520234cf9b24a306829ecd80953cca87b6d6280f19e011cc8c49a784507a7f: Status 404 returned error can't find the container with id 3f520234cf9b24a306829ecd80953cca87b6d6280f19e011cc8c49a784507a7f
Sep 09 16:39:12.036023 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:12.035987    2576 generic.go:356] "Generic (PLEG): container finished" podID="11313a0b-f798-4005-954b-fd2628ff668f" containerID="af25a25597510754815845c528c605cf8da2f085a651f8c9377b6ef64e0e2102" exitCode=0
Sep 09 16:39:12.036408 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:12.036039    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68" event={"ID":"11313a0b-f798-4005-954b-fd2628ff668f","Type":"ContainerDied","Data":"af25a25597510754815845c528c605cf8da2f085a651f8c9377b6ef64e0e2102"}
Sep 09 16:39:12.036408 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:12.036091    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68" event={"ID":"11313a0b-f798-4005-954b-fd2628ff668f","Type":"ContainerStarted","Data":"3f520234cf9b24a306829ecd80953cca87b6d6280f19e011cc8c49a784507a7f"}
Sep 09 16:39:13.170051 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:13.170030    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68"
Sep 09 16:39:13.273187 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:13.273158    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11313a0b-f798-4005-954b-fd2628ff668f-maas-api-service-ca\") pod \"11313a0b-f798-4005-954b-fd2628ff668f\" (UID: \"11313a0b-f798-4005-954b-fd2628ff668f\") "
Sep 09 16:39:13.273507 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:13.273486    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11313a0b-f798-4005-954b-fd2628ff668f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "11313a0b-f798-4005-954b-fd2628ff668f" (UID: "11313a0b-f798-4005-954b-fd2628ff668f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:39:13.374515 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:13.374435    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11313a0b-f798-4005-954b-fd2628ff668f-maas-api-service-ca\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:39:14.046785 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:14.046758    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68"
Sep 09 16:39:14.046980 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:14.046749    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68" event={"ID":"11313a0b-f798-4005-954b-fd2628ff668f","Type":"ContainerDied","Data":"3f520234cf9b24a306829ecd80953cca87b6d6280f19e011cc8c49a784507a7f"}
Sep 09 16:39:14.046980 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:14.046926    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f520234cf9b24a306829ecd80953cca87b6d6280f19e011cc8c49a784507a7f"
Sep 09 16:39:16.604995 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:16.604958    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"]
Sep 09 16:39:16.605472 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:16.605228    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6" podUID="465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f" containerName="payload-processing" containerID="cri-o://efcd3bb5c8e15a9747ae3ee11956b3516528aaf0054f418bc1fca5314b7e0770" gracePeriod=30
Sep 09 16:39:16.613906 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:16.613862    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"]
Sep 09 16:39:16.614123 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:16.614093    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt" podUID="7eb455ef-0b5b-42f5-8e01-dc84246dd8e0" containerName="payload-pre-processing" containerID="cri-o://ad33524bdedf00457ccd78d4ce61ec9cc218af8a9581aa77a52d45627ea18637" gracePeriod=30
Sep 09 16:39:16.781483 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:16.781413    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68"]
Sep 09 16:39:16.784041 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:16.784017    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-936acc-rxs68"]
Sep 09 16:39:16.878893 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:16.878871    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"
Sep 09 16:39:16.881787 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:16.881769    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"
Sep 09 16:39:17.013785 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.013741    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f-plugins-config-volume\") pod \"465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f\" (UID: \"465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f\") "
Sep 09 16:39:17.013970 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.013816    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v857\" (UniqueName: \"kubernetes.io/projected/7eb455ef-0b5b-42f5-8e01-dc84246dd8e0-kube-api-access-2v857\") pod \"7eb455ef-0b5b-42f5-8e01-dc84246dd8e0\" (UID: \"7eb455ef-0b5b-42f5-8e01-dc84246dd8e0\") "
Sep 09 16:39:17.013970 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.013841    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgqbj\" (UniqueName: \"kubernetes.io/projected/465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f-kube-api-access-dgqbj\") pod \"465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f\" (UID: \"465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f\") "
Sep 09 16:39:17.013970 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.013916    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eb455ef-0b5b-42f5-8e01-dc84246dd8e0-plugins-config-volume\") pod \"7eb455ef-0b5b-42f5-8e01-dc84246dd8e0\" (UID: \"7eb455ef-0b5b-42f5-8e01-dc84246dd8e0\") "
Sep 09 16:39:17.014291 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.014164    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f" (UID: "465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:39:17.014418 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.014341    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb455ef-0b5b-42f5-8e01-dc84246dd8e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7eb455ef-0b5b-42f5-8e01-dc84246dd8e0" (UID: "7eb455ef-0b5b-42f5-8e01-dc84246dd8e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:39:17.016221 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.016198    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f-kube-api-access-dgqbj" (OuterVolumeSpecName: "kube-api-access-dgqbj") pod "465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f" (UID: "465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f"). InnerVolumeSpecName "kube-api-access-dgqbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:39:17.016309 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.016242    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb455ef-0b5b-42f5-8e01-dc84246dd8e0-kube-api-access-2v857" (OuterVolumeSpecName: "kube-api-access-2v857") pod "7eb455ef-0b5b-42f5-8e01-dc84246dd8e0" (UID: "7eb455ef-0b5b-42f5-8e01-dc84246dd8e0"). InnerVolumeSpecName "kube-api-access-2v857". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:39:17.061434 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.061395    2576 generic.go:356] "Generic (PLEG): container finished" podID="7eb455ef-0b5b-42f5-8e01-dc84246dd8e0" containerID="ad33524bdedf00457ccd78d4ce61ec9cc218af8a9581aa77a52d45627ea18637" exitCode=0
Sep 09 16:39:17.061631 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.061443    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"
Sep 09 16:39:17.061631 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.061495    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt" event={"ID":"7eb455ef-0b5b-42f5-8e01-dc84246dd8e0","Type":"ContainerDied","Data":"ad33524bdedf00457ccd78d4ce61ec9cc218af8a9581aa77a52d45627ea18637"}
Sep 09 16:39:17.061631 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.061546    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt" event={"ID":"7eb455ef-0b5b-42f5-8e01-dc84246dd8e0","Type":"ContainerDied","Data":"164ebc29be47a5b50441fc0c01340088d0caf42f5be21bf24f80c5afe1984b35"}
Sep 09 16:39:17.061631 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.061573    2576 scope.go:117] "RemoveContainer" containerID="ad33524bdedf00457ccd78d4ce61ec9cc218af8a9581aa77a52d45627ea18637"
Sep 09 16:39:17.062891 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.062870    2576 generic.go:356] "Generic (PLEG): container finished" podID="465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f" containerID="efcd3bb5c8e15a9747ae3ee11956b3516528aaf0054f418bc1fca5314b7e0770" exitCode=0
Sep 09 16:39:17.062961 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.062896    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"
Sep 09 16:39:17.062961 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.062941    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6" event={"ID":"465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f","Type":"ContainerDied","Data":"efcd3bb5c8e15a9747ae3ee11956b3516528aaf0054f418bc1fca5314b7e0770"}
Sep 09 16:39:17.063038 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.062983    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6" event={"ID":"465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f","Type":"ContainerDied","Data":"b2b3c63b838c990f3a2e5b637b664a5d97285098577f9be1e8704375c1ff171b"}
Sep 09 16:39:17.070923 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.070905    2576 scope.go:117] "RemoveContainer" containerID="ad33524bdedf00457ccd78d4ce61ec9cc218af8a9581aa77a52d45627ea18637"
Sep 09 16:39:17.071215 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:39:17.071197    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad33524bdedf00457ccd78d4ce61ec9cc218af8a9581aa77a52d45627ea18637\": container with ID starting with ad33524bdedf00457ccd78d4ce61ec9cc218af8a9581aa77a52d45627ea18637 not found: ID does not exist" containerID="ad33524bdedf00457ccd78d4ce61ec9cc218af8a9581aa77a52d45627ea18637"
Sep 09 16:39:17.071288 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.071225    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad33524bdedf00457ccd78d4ce61ec9cc218af8a9581aa77a52d45627ea18637"} err="failed to get container status \"ad33524bdedf00457ccd78d4ce61ec9cc218af8a9581aa77a52d45627ea18637\": rpc error: code = NotFound desc = could not find container \"ad33524bdedf00457ccd78d4ce61ec9cc218af8a9581aa77a52d45627ea18637\": container with ID starting with ad33524bdedf00457ccd78d4ce61ec9cc218af8a9581aa77a52d45627ea18637 not found: ID does not exist"
Sep 09 16:39:17.071288 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.071241    2576 scope.go:117] "RemoveContainer" containerID="efcd3bb5c8e15a9747ae3ee11956b3516528aaf0054f418bc1fca5314b7e0770"
Sep 09 16:39:17.079126 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.079109    2576 scope.go:117] "RemoveContainer" containerID="efcd3bb5c8e15a9747ae3ee11956b3516528aaf0054f418bc1fca5314b7e0770"
Sep 09 16:39:17.079388 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:39:17.079367    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efcd3bb5c8e15a9747ae3ee11956b3516528aaf0054f418bc1fca5314b7e0770\": container with ID starting with efcd3bb5c8e15a9747ae3ee11956b3516528aaf0054f418bc1fca5314b7e0770 not found: ID does not exist" containerID="efcd3bb5c8e15a9747ae3ee11956b3516528aaf0054f418bc1fca5314b7e0770"
Sep 09 16:39:17.079443 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.079396    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efcd3bb5c8e15a9747ae3ee11956b3516528aaf0054f418bc1fca5314b7e0770"} err="failed to get container status \"efcd3bb5c8e15a9747ae3ee11956b3516528aaf0054f418bc1fca5314b7e0770\": rpc error: code = NotFound desc = could not find container \"efcd3bb5c8e15a9747ae3ee11956b3516528aaf0054f418bc1fca5314b7e0770\": container with ID starting with efcd3bb5c8e15a9747ae3ee11956b3516528aaf0054f418bc1fca5314b7e0770 not found: ID does not exist"
Sep 09 16:39:17.085446 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.085423    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"]
Sep 09 16:39:17.090220 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.090201    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-936acc-5764796966-bbngt"]
Sep 09 16:39:17.100932 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.100914    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"]
Sep 09 16:39:17.104722 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.104691    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-936acc-74fcfc85df-fwnl6"]
Sep 09 16:39:17.114831 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.114784    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eb455ef-0b5b-42f5-8e01-dc84246dd8e0-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:39:17.114831 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.114807    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:39:17.114831 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.114818    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v857\" (UniqueName: \"kubernetes.io/projected/7eb455ef-0b5b-42f5-8e01-dc84246dd8e0-kube-api-access-2v857\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:39:17.114959 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:17.114828    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgqbj\" (UniqueName: \"kubernetes.io/projected/465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f-kube-api-access-dgqbj\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:39:18.553026 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:18.552983    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11313a0b-f798-4005-954b-fd2628ff668f" path="/var/lib/kubelet/pods/11313a0b-f798-4005-954b-fd2628ff668f/volumes"
Sep 09 16:39:18.553398 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:18.553350    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f" path="/var/lib/kubelet/pods/465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f/volumes"
Sep 09 16:39:18.553647 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:18.553635    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eb455ef-0b5b-42f5-8e01-dc84246dd8e0" path="/var/lib/kubelet/pods/7eb455ef-0b5b-42f5-8e01-dc84246dd8e0/volumes"
Sep 09 16:39:27.127164 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.127128    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k"]
Sep 09 16:39:27.127542 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.127438    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11313a0b-f798-4005-954b-fd2628ff668f" containerName="revoke-keys"
Sep 09 16:39:27.127542 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.127448    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="11313a0b-f798-4005-954b-fd2628ff668f" containerName="revoke-keys"
Sep 09 16:39:27.127542 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.127460    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f" containerName="payload-processing"
Sep 09 16:39:27.127542 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.127466    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f" containerName="payload-processing"
Sep 09 16:39:27.127542 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.127485    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb455ef-0b5b-42f5-8e01-dc84246dd8e0" containerName="payload-pre-processing"
Sep 09 16:39:27.127542 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.127491    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb455ef-0b5b-42f5-8e01-dc84246dd8e0" containerName="payload-pre-processing"
Sep 09 16:39:27.127815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.127557    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eb455ef-0b5b-42f5-8e01-dc84246dd8e0" containerName="payload-pre-processing"
Sep 09 16:39:27.127815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.127566    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="11313a0b-f798-4005-954b-fd2628ff668f" containerName="revoke-keys"
Sep 09 16:39:27.127815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.127572    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="465bd2a2-fbb2-45eb-9d7a-49a5e0f75c2f" containerName="payload-processing"
Sep 09 16:39:27.131554 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.131538    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k"
Sep 09 16:39:27.134477 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.134454    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxghs\""
Sep 09 16:39:27.134709 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.134519    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:39:27.137796 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.137772    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k"]
Sep 09 16:39:27.303125 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.303092    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b2e5eb-fe2c-46f7-8de6-b44d62eaab72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k\" (UID: \"52b2e5eb-fe2c-46f7-8de6-b44d62eaab72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k"
Sep 09 16:39:27.404164 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.404067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b2e5eb-fe2c-46f7-8de6-b44d62eaab72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k\" (UID: \"52b2e5eb-fe2c-46f7-8de6-b44d62eaab72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k"
Sep 09 16:39:27.404659 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.404639    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b2e5eb-fe2c-46f7-8de6-b44d62eaab72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k\" (UID: \"52b2e5eb-fe2c-46f7-8de6-b44d62eaab72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k"
Sep 09 16:39:27.443368 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.443346    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k"
Sep 09 16:39:27.710800 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:27.710777    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k"]
Sep 09 16:39:27.713251 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:39:27.713226    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b2e5eb_fe2c_46f7_8de6_b44d62eaab72.slice/crio-09ffe6e5a246f925e372428295a8641afc4cd17f72c634aa6df8255978eca40a WatchSource:0}: Error finding container 09ffe6e5a246f925e372428295a8641afc4cd17f72c634aa6df8255978eca40a: Status 404 returned error can't find the container with id 09ffe6e5a246f925e372428295a8641afc4cd17f72c634aa6df8255978eca40a
Sep 09 16:39:28.116648 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:28.116616    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k" event={"ID":"52b2e5eb-fe2c-46f7-8de6-b44d62eaab72","Type":"ContainerStarted","Data":"b897931c6cb6c83834600c90f79aef7e9ea6c967a6decc2580f42a3ac0ee25a9"}
Sep 09 16:39:28.116823 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:28.116664    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k" event={"ID":"52b2e5eb-fe2c-46f7-8de6-b44d62eaab72","Type":"ContainerStarted","Data":"09ffe6e5a246f925e372428295a8641afc4cd17f72c634aa6df8255978eca40a"}
Sep 09 16:39:28.132492 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:28.132443    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k" podStartSLOduration=1.1324271320000001 podStartE2EDuration="1.132427132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:39:28.13119792 +0000 UTC m=+1844.066816350" watchObservedRunningTime="2026-09-09 16:39:28.132427132 +0000 UTC m=+1844.068045579"
Sep 09 16:39:29.122558 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:29.122525    2576 generic.go:356] "Generic (PLEG): container finished" podID="52b2e5eb-fe2c-46f7-8de6-b44d62eaab72" containerID="b897931c6cb6c83834600c90f79aef7e9ea6c967a6decc2580f42a3ac0ee25a9" exitCode=0
Sep 09 16:39:29.122774 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:29.122618    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k" event={"ID":"52b2e5eb-fe2c-46f7-8de6-b44d62eaab72","Type":"ContainerDied","Data":"b897931c6cb6c83834600c90f79aef7e9ea6c967a6decc2580f42a3ac0ee25a9"}
Sep 09 16:39:30.255853 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:30.255826    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k"
Sep 09 16:39:30.331244 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:30.331212    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b2e5eb-fe2c-46f7-8de6-b44d62eaab72-maas-api-service-ca\") pod \"52b2e5eb-fe2c-46f7-8de6-b44d62eaab72\" (UID: \"52b2e5eb-fe2c-46f7-8de6-b44d62eaab72\") "
Sep 09 16:39:30.331590 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:30.331561    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b2e5eb-fe2c-46f7-8de6-b44d62eaab72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52b2e5eb-fe2c-46f7-8de6-b44d62eaab72" (UID: "52b2e5eb-fe2c-46f7-8de6-b44d62eaab72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:39:30.431997 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:30.431911    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b2e5eb-fe2c-46f7-8de6-b44d62eaab72-maas-api-service-ca\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:39:31.133246 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:31.133211    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k"
Sep 09 16:39:31.133246 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:31.133232    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k" event={"ID":"52b2e5eb-fe2c-46f7-8de6-b44d62eaab72","Type":"ContainerDied","Data":"09ffe6e5a246f925e372428295a8641afc4cd17f72c634aa6df8255978eca40a"}
Sep 09 16:39:31.133474 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:31.133288    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09ffe6e5a246f925e372428295a8641afc4cd17f72c634aa6df8255978eca40a"
Sep 09 16:39:42.799591 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:42.799559    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k"]
Sep 09 16:39:42.801146 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:42.801122    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54370d-ts94k"]
Sep 09 16:39:44.553510 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:44.553477    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52b2e5eb-fe2c-46f7-8de6-b44d62eaab72" path="/var/lib/kubelet/pods/52b2e5eb-fe2c-46f7-8de6-b44d62eaab72/volumes"
Sep 09 16:39:48.200420 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:39:48.200385    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"]
Sep 09 16:40:03.221836 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.221793    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd" podUID="77abe4c7-6a16-41b6-b0c9-c28535beae96" containerName="main" containerID="cri-o://24df2430b20c1aec9bf8605770e6ef38b1658f0b63d25310c6ef1a0a08213456" gracePeriod=45
Sep 09 16:40:03.471912 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.471847    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:40:03.617853 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.617807    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-dshm\") pod \"77abe4c7-6a16-41b6-b0c9-c28535beae96\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") "
Sep 09 16:40:03.618055 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.617864    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-home\") pod \"77abe4c7-6a16-41b6-b0c9-c28535beae96\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") "
Sep 09 16:40:03.618055 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.617907    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-tmp-dir\") pod \"77abe4c7-6a16-41b6-b0c9-c28535beae96\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") "
Sep 09 16:40:03.618055 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.617931    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhfk2\" (UniqueName: \"kubernetes.io/projected/77abe4c7-6a16-41b6-b0c9-c28535beae96-kube-api-access-jhfk2\") pod \"77abe4c7-6a16-41b6-b0c9-c28535beae96\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") "
Sep 09 16:40:03.618055 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.617967    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77abe4c7-6a16-41b6-b0c9-c28535beae96-tls-certs\") pod \"77abe4c7-6a16-41b6-b0c9-c28535beae96\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") "
Sep 09 16:40:03.618055 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.618013    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-model-cache\") pod \"77abe4c7-6a16-41b6-b0c9-c28535beae96\" (UID: \"77abe4c7-6a16-41b6-b0c9-c28535beae96\") "
Sep 09 16:40:03.618339 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.618200    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-home" (OuterVolumeSpecName: "home") pod "77abe4c7-6a16-41b6-b0c9-c28535beae96" (UID: "77abe4c7-6a16-41b6-b0c9-c28535beae96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:40:03.618399 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.618346    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-model-cache" (OuterVolumeSpecName: "model-cache") pod "77abe4c7-6a16-41b6-b0c9-c28535beae96" (UID: "77abe4c7-6a16-41b6-b0c9-c28535beae96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:40:03.618399 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.618365    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77abe4c7-6a16-41b6-b0c9-c28535beae96" (UID: "77abe4c7-6a16-41b6-b0c9-c28535beae96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:40:03.620239 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.620217    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-dshm" (OuterVolumeSpecName: "dshm") pod "77abe4c7-6a16-41b6-b0c9-c28535beae96" (UID: "77abe4c7-6a16-41b6-b0c9-c28535beae96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:40:03.620338 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.620253    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77abe4c7-6a16-41b6-b0c9-c28535beae96-kube-api-access-jhfk2" (OuterVolumeSpecName: "kube-api-access-jhfk2") pod "77abe4c7-6a16-41b6-b0c9-c28535beae96" (UID: "77abe4c7-6a16-41b6-b0c9-c28535beae96"). InnerVolumeSpecName "kube-api-access-jhfk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:40:03.620338 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.620315    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77abe4c7-6a16-41b6-b0c9-c28535beae96-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77abe4c7-6a16-41b6-b0c9-c28535beae96" (UID: "77abe4c7-6a16-41b6-b0c9-c28535beae96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:40:03.719305 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.719268    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:03.719305 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.719306    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhfk2\" (UniqueName: \"kubernetes.io/projected/77abe4c7-6a16-41b6-b0c9-c28535beae96-kube-api-access-jhfk2\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:03.719567 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.719320    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77abe4c7-6a16-41b6-b0c9-c28535beae96-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:03.719567 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.719333    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:03.719567 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.719345    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:03.719567 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:03.719356    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77abe4c7-6a16-41b6-b0c9-c28535beae96-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:04.293286 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:04.293245    2576 generic.go:356] "Generic (PLEG): container finished" podID="77abe4c7-6a16-41b6-b0c9-c28535beae96" containerID="24df2430b20c1aec9bf8605770e6ef38b1658f0b63d25310c6ef1a0a08213456" exitCode=0
Sep 09 16:40:04.293762 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:04.293308    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"
Sep 09 16:40:04.293762 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:04.293325    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd" event={"ID":"77abe4c7-6a16-41b6-b0c9-c28535beae96","Type":"ContainerDied","Data":"24df2430b20c1aec9bf8605770e6ef38b1658f0b63d25310c6ef1a0a08213456"}
Sep 09 16:40:04.293762 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:04.293369    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd" event={"ID":"77abe4c7-6a16-41b6-b0c9-c28535beae96","Type":"ContainerDied","Data":"77b57225d58dbaa65ab6a4e14cac61a6f69631d3d844d6995246e56fbf061e61"}
Sep 09 16:40:04.293762 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:04.293386    2576 scope.go:117] "RemoveContainer" containerID="24df2430b20c1aec9bf8605770e6ef38b1658f0b63d25310c6ef1a0a08213456"
Sep 09 16:40:04.305403 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:04.305375    2576 scope.go:117] "RemoveContainer" containerID="24df2430b20c1aec9bf8605770e6ef38b1658f0b63d25310c6ef1a0a08213456"
Sep 09 16:40:04.305768 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:40:04.305744    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24df2430b20c1aec9bf8605770e6ef38b1658f0b63d25310c6ef1a0a08213456\": container with ID starting with 24df2430b20c1aec9bf8605770e6ef38b1658f0b63d25310c6ef1a0a08213456 not found: ID does not exist" containerID="24df2430b20c1aec9bf8605770e6ef38b1658f0b63d25310c6ef1a0a08213456"
Sep 09 16:40:04.305843 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:04.305783    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24df2430b20c1aec9bf8605770e6ef38b1658f0b63d25310c6ef1a0a08213456"} err="failed to get container status \"24df2430b20c1aec9bf8605770e6ef38b1658f0b63d25310c6ef1a0a08213456\": rpc error: code = NotFound desc = could not find container \"24df2430b20c1aec9bf8605770e6ef38b1658f0b63d25310c6ef1a0a08213456\": container with ID starting with 24df2430b20c1aec9bf8605770e6ef38b1658f0b63d25310c6ef1a0a08213456 not found: ID does not exist"
Sep 09 16:40:04.315142 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:04.315117    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"]
Sep 09 16:40:04.320681 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:04.320652    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-54370d/ipp-route-54370d-kserve-d6bbd9767-2b7wd"]
Sep 09 16:40:04.554550 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:04.554457    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77abe4c7-6a16-41b6-b0c9-c28535beae96" path="/var/lib/kubelet/pods/77abe4c7-6a16-41b6-b0c9-c28535beae96/volumes"
Sep 09 16:40:31.296737 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.296685    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk"]
Sep 09 16:40:31.297109 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.297036    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77abe4c7-6a16-41b6-b0c9-c28535beae96" containerName="main"
Sep 09 16:40:31.297109 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.297048    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="77abe4c7-6a16-41b6-b0c9-c28535beae96" containerName="main"
Sep 09 16:40:31.297109 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.297056    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b2e5eb-fe2c-46f7-8de6-b44d62eaab72" containerName="revoke-keys"
Sep 09 16:40:31.297109 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.297061    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b2e5eb-fe2c-46f7-8de6-b44d62eaab72" containerName="revoke-keys"
Sep 09 16:40:31.297264 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.297126    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="77abe4c7-6a16-41b6-b0c9-c28535beae96" containerName="main"
Sep 09 16:40:31.297264 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.297138    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="52b2e5eb-fe2c-46f7-8de6-b44d62eaab72" containerName="revoke-keys"
Sep 09 16:40:31.299822 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.299806    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk"
Sep 09 16:40:31.302330 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.302306    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:40:31.302467 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.302349    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxghs\""
Sep 09 16:40:31.307183 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.307016    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk"]
Sep 09 16:40:31.327041 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.327016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a83e2d-d5d8-4ea4-87f7-76ac6a815133-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk\" (UID: \"69a83e2d-d5d8-4ea4-87f7-76ac6a815133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk"
Sep 09 16:40:31.427865 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.427827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a83e2d-d5d8-4ea4-87f7-76ac6a815133-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk\" (UID: \"69a83e2d-d5d8-4ea4-87f7-76ac6a815133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk"
Sep 09 16:40:31.428409 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.428391    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a83e2d-d5d8-4ea4-87f7-76ac6a815133-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk\" (UID: \"69a83e2d-d5d8-4ea4-87f7-76ac6a815133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk"
Sep 09 16:40:31.611343 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.611254    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk"
Sep 09 16:40:31.880901 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:31.880814    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk"]
Sep 09 16:40:31.883762 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:40:31.883730    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a83e2d_d5d8_4ea4_87f7_76ac6a815133.slice/crio-5bc22f32d904fffca939c76b33514ed3fda117e69f2980351330f4e86950db11 WatchSource:0}: Error finding container 5bc22f32d904fffca939c76b33514ed3fda117e69f2980351330f4e86950db11: Status 404 returned error can't find the container with id 5bc22f32d904fffca939c76b33514ed3fda117e69f2980351330f4e86950db11
Sep 09 16:40:32.422090 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:32.422055    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk" event={"ID":"69a83e2d-d5d8-4ea4-87f7-76ac6a815133","Type":"ContainerStarted","Data":"9cc3e3971297e0a80d73f47d15f78f7d5b44861341f7ffc36eba9a424320bbb0"}
Sep 09 16:40:32.422478 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:32.422110    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk" event={"ID":"69a83e2d-d5d8-4ea4-87f7-76ac6a815133","Type":"ContainerStarted","Data":"5bc22f32d904fffca939c76b33514ed3fda117e69f2980351330f4e86950db11"}
Sep 09 16:40:32.447320 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:32.447263    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk" podStartSLOduration=1.4472495 podStartE2EDuration="1.4472495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:40:32.446456907 +0000 UTC m=+1908.382075334" watchObservedRunningTime="2026-09-09 16:40:32.4472495 +0000 UTC m=+1908.382867915"
Sep 09 16:40:33.430936 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:33.430904    2576 generic.go:356] "Generic (PLEG): container finished" podID="69a83e2d-d5d8-4ea4-87f7-76ac6a815133" containerID="9cc3e3971297e0a80d73f47d15f78f7d5b44861341f7ffc36eba9a424320bbb0" exitCode=0
Sep 09 16:40:33.431352 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:33.430993    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk" event={"ID":"69a83e2d-d5d8-4ea4-87f7-76ac6a815133","Type":"ContainerDied","Data":"9cc3e3971297e0a80d73f47d15f78f7d5b44861341f7ffc36eba9a424320bbb0"}
Sep 09 16:40:34.569305 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:34.569285    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk"
Sep 09 16:40:34.653281 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:34.653246    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a83e2d-d5d8-4ea4-87f7-76ac6a815133-maas-api-service-ca\") pod \"69a83e2d-d5d8-4ea4-87f7-76ac6a815133\" (UID: \"69a83e2d-d5d8-4ea4-87f7-76ac6a815133\") "
Sep 09 16:40:34.653567 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:34.653543    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a83e2d-d5d8-4ea4-87f7-76ac6a815133-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69a83e2d-d5d8-4ea4-87f7-76ac6a815133" (UID: "69a83e2d-d5d8-4ea4-87f7-76ac6a815133"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:40:34.754237 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:34.754208    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a83e2d-d5d8-4ea4-87f7-76ac6a815133-maas-api-service-ca\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:35.442511 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:35.442485    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk"
Sep 09 16:40:35.442723 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:35.442525    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk" event={"ID":"69a83e2d-d5d8-4ea4-87f7-76ac6a815133","Type":"ContainerDied","Data":"5bc22f32d904fffca939c76b33514ed3fda117e69f2980351330f4e86950db11"}
Sep 09 16:40:35.442723 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:35.442580    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bc22f32d904fffca939c76b33514ed3fda117e69f2980351330f4e86950db11"
Sep 09 16:40:41.788036 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:41.787999    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"]
Sep 09 16:40:41.788475 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:41.788237    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw" podUID="e33c68b8-749c-4a69-96b6-cdc447af2192" containerName="payload-processing" containerID="cri-o://2cb60b46acaa4692f9da93918192cb379e441de2a0df7c1cef0488e8294ca5c7" gracePeriod=30
Sep 09 16:40:41.798927 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:41.798899    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"]
Sep 09 16:40:41.799101 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:41.799082    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q" podUID="9bf403f4-0c5d-4c72-b002-b4fbee774bc3" containerName="payload-pre-processing" containerID="cri-o://ce72aa77a2347122175e1f050fb003af28e19c1bc162e6b12dfae70d1a64cf81" gracePeriod=30
Sep 09 16:40:41.952336 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:41.952297    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk"]
Sep 09 16:40:41.957174 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:41.957144    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c6036-rdnxk"]
Sep 09 16:40:42.051325 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.051268    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"
Sep 09 16:40:42.083599 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.083576    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"
Sep 09 16:40:42.116569 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.116545    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33c68b8-749c-4a69-96b6-cdc447af2192-plugins-config-volume\") pod \"e33c68b8-749c-4a69-96b6-cdc447af2192\" (UID: \"e33c68b8-749c-4a69-96b6-cdc447af2192\") "
Sep 09 16:40:42.116756 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.116636    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2swr2\" (UniqueName: \"kubernetes.io/projected/9bf403f4-0c5d-4c72-b002-b4fbee774bc3-kube-api-access-2swr2\") pod \"9bf403f4-0c5d-4c72-b002-b4fbee774bc3\" (UID: \"9bf403f4-0c5d-4c72-b002-b4fbee774bc3\") "
Sep 09 16:40:42.116756 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.116660    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dhzk\" (UniqueName: \"kubernetes.io/projected/e33c68b8-749c-4a69-96b6-cdc447af2192-kube-api-access-8dhzk\") pod \"e33c68b8-749c-4a69-96b6-cdc447af2192\" (UID: \"e33c68b8-749c-4a69-96b6-cdc447af2192\") "
Sep 09 16:40:42.116756 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.116724    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf403f4-0c5d-4c72-b002-b4fbee774bc3-plugins-config-volume\") pod \"9bf403f4-0c5d-4c72-b002-b4fbee774bc3\" (UID: \"9bf403f4-0c5d-4c72-b002-b4fbee774bc3\") "
Sep 09 16:40:42.117069 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.117038    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e33c68b8-749c-4a69-96b6-cdc447af2192-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e33c68b8-749c-4a69-96b6-cdc447af2192" (UID: "e33c68b8-749c-4a69-96b6-cdc447af2192"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:40:42.117207 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.117175    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf403f4-0c5d-4c72-b002-b4fbee774bc3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9bf403f4-0c5d-4c72-b002-b4fbee774bc3" (UID: "9bf403f4-0c5d-4c72-b002-b4fbee774bc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:40:42.118883 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.118861    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf403f4-0c5d-4c72-b002-b4fbee774bc3-kube-api-access-2swr2" (OuterVolumeSpecName: "kube-api-access-2swr2") pod "9bf403f4-0c5d-4c72-b002-b4fbee774bc3" (UID: "9bf403f4-0c5d-4c72-b002-b4fbee774bc3"). InnerVolumeSpecName "kube-api-access-2swr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:40:42.118959 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.118939    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33c68b8-749c-4a69-96b6-cdc447af2192-kube-api-access-8dhzk" (OuterVolumeSpecName: "kube-api-access-8dhzk") pod "e33c68b8-749c-4a69-96b6-cdc447af2192" (UID: "e33c68b8-749c-4a69-96b6-cdc447af2192"). InnerVolumeSpecName "kube-api-access-8dhzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:40:42.218321 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.218286    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf403f4-0c5d-4c72-b002-b4fbee774bc3-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:42.218321 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.218313    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33c68b8-749c-4a69-96b6-cdc447af2192-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:42.218321 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.218322    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2swr2\" (UniqueName: \"kubernetes.io/projected/9bf403f4-0c5d-4c72-b002-b4fbee774bc3-kube-api-access-2swr2\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:42.218551 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.218331    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dhzk\" (UniqueName: \"kubernetes.io/projected/e33c68b8-749c-4a69-96b6-cdc447af2192-kube-api-access-8dhzk\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:42.477921 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.477882    2576 generic.go:356] "Generic (PLEG): container finished" podID="9bf403f4-0c5d-4c72-b002-b4fbee774bc3" containerID="ce72aa77a2347122175e1f050fb003af28e19c1bc162e6b12dfae70d1a64cf81" exitCode=0
Sep 09 16:40:42.478096 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.477933    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"
Sep 09 16:40:42.478096 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.477980    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q" event={"ID":"9bf403f4-0c5d-4c72-b002-b4fbee774bc3","Type":"ContainerDied","Data":"ce72aa77a2347122175e1f050fb003af28e19c1bc162e6b12dfae70d1a64cf81"}
Sep 09 16:40:42.478096 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.478068    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q" event={"ID":"9bf403f4-0c5d-4c72-b002-b4fbee774bc3","Type":"ContainerDied","Data":"4dad803e89cb6cd2b423376c21617a23d23b96cb0f7ed7776cc71264e9a38800"}
Sep 09 16:40:42.478096 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.478096    2576 scope.go:117] "RemoveContainer" containerID="ce72aa77a2347122175e1f050fb003af28e19c1bc162e6b12dfae70d1a64cf81"
Sep 09 16:40:42.479479 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.479458    2576 generic.go:356] "Generic (PLEG): container finished" podID="e33c68b8-749c-4a69-96b6-cdc447af2192" containerID="2cb60b46acaa4692f9da93918192cb379e441de2a0df7c1cef0488e8294ca5c7" exitCode=0
Sep 09 16:40:42.479588 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.479500    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw" event={"ID":"e33c68b8-749c-4a69-96b6-cdc447af2192","Type":"ContainerDied","Data":"2cb60b46acaa4692f9da93918192cb379e441de2a0df7c1cef0488e8294ca5c7"}
Sep 09 16:40:42.479588 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.479541    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw" event={"ID":"e33c68b8-749c-4a69-96b6-cdc447af2192","Type":"ContainerDied","Data":"b4b81c4841b9ed522524ebaee19f438bdaa18730e6cda845bf9bb1746d8ceb29"}
Sep 09 16:40:42.479588 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.479502    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"
Sep 09 16:40:42.487728 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.487687    2576 scope.go:117] "RemoveContainer" containerID="ce72aa77a2347122175e1f050fb003af28e19c1bc162e6b12dfae70d1a64cf81"
Sep 09 16:40:42.488015 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:40:42.487991    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce72aa77a2347122175e1f050fb003af28e19c1bc162e6b12dfae70d1a64cf81\": container with ID starting with ce72aa77a2347122175e1f050fb003af28e19c1bc162e6b12dfae70d1a64cf81 not found: ID does not exist" containerID="ce72aa77a2347122175e1f050fb003af28e19c1bc162e6b12dfae70d1a64cf81"
Sep 09 16:40:42.488081 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.488017    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce72aa77a2347122175e1f050fb003af28e19c1bc162e6b12dfae70d1a64cf81"} err="failed to get container status \"ce72aa77a2347122175e1f050fb003af28e19c1bc162e6b12dfae70d1a64cf81\": rpc error: code = NotFound desc = could not find container \"ce72aa77a2347122175e1f050fb003af28e19c1bc162e6b12dfae70d1a64cf81\": container with ID starting with ce72aa77a2347122175e1f050fb003af28e19c1bc162e6b12dfae70d1a64cf81 not found: ID does not exist"
Sep 09 16:40:42.488081 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.488035    2576 scope.go:117] "RemoveContainer" containerID="2cb60b46acaa4692f9da93918192cb379e441de2a0df7c1cef0488e8294ca5c7"
Sep 09 16:40:42.496160 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.496146    2576 scope.go:117] "RemoveContainer" containerID="2cb60b46acaa4692f9da93918192cb379e441de2a0df7c1cef0488e8294ca5c7"
Sep 09 16:40:42.496402 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:40:42.496384    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb60b46acaa4692f9da93918192cb379e441de2a0df7c1cef0488e8294ca5c7\": container with ID starting with 2cb60b46acaa4692f9da93918192cb379e441de2a0df7c1cef0488e8294ca5c7 not found: ID does not exist" containerID="2cb60b46acaa4692f9da93918192cb379e441de2a0df7c1cef0488e8294ca5c7"
Sep 09 16:40:42.496465 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.496413    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb60b46acaa4692f9da93918192cb379e441de2a0df7c1cef0488e8294ca5c7"} err="failed to get container status \"2cb60b46acaa4692f9da93918192cb379e441de2a0df7c1cef0488e8294ca5c7\": rpc error: code = NotFound desc = could not find container \"2cb60b46acaa4692f9da93918192cb379e441de2a0df7c1cef0488e8294ca5c7\": container with ID starting with 2cb60b46acaa4692f9da93918192cb379e441de2a0df7c1cef0488e8294ca5c7 not found: ID does not exist"
Sep 09 16:40:42.502578 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.502558    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"]
Sep 09 16:40:42.506317 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.506298    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c6036-869f766d54-j427q"]
Sep 09 16:40:42.518015 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.517996    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"]
Sep 09 16:40:42.521017 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.520999    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c6036-77bfd75c85-rzthw"]
Sep 09 16:40:42.552598 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.552569    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a83e2d-d5d8-4ea4-87f7-76ac6a815133" path="/var/lib/kubelet/pods/69a83e2d-d5d8-4ea4-87f7-76ac6a815133/volumes"
Sep 09 16:40:42.552941 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.552927    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bf403f4-0c5d-4c72-b002-b4fbee774bc3" path="/var/lib/kubelet/pods/9bf403f4-0c5d-4c72-b002-b4fbee774bc3/volumes"
Sep 09 16:40:42.553244 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:42.553232    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e33c68b8-749c-4a69-96b6-cdc447af2192" path="/var/lib/kubelet/pods/e33c68b8-749c-4a69-96b6-cdc447af2192/volumes"
Sep 09 16:40:48.851129 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.851093    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4"]
Sep 09 16:40:48.851498 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.851392    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a83e2d-d5d8-4ea4-87f7-76ac6a815133" containerName="revoke-keys"
Sep 09 16:40:48.851498 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.851403    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a83e2d-d5d8-4ea4-87f7-76ac6a815133" containerName="revoke-keys"
Sep 09 16:40:48.851498 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.851415    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bf403f4-0c5d-4c72-b002-b4fbee774bc3" containerName="payload-pre-processing"
Sep 09 16:40:48.851498 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.851421    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bf403f4-0c5d-4c72-b002-b4fbee774bc3" containerName="payload-pre-processing"
Sep 09 16:40:48.851498 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.851428    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33c68b8-749c-4a69-96b6-cdc447af2192" containerName="payload-processing"
Sep 09 16:40:48.851498 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.851435    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33c68b8-749c-4a69-96b6-cdc447af2192" containerName="payload-processing"
Sep 09 16:40:48.851498 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.851487    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bf403f4-0c5d-4c72-b002-b4fbee774bc3" containerName="payload-pre-processing"
Sep 09 16:40:48.851498 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.851497    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a83e2d-d5d8-4ea4-87f7-76ac6a815133" containerName="revoke-keys"
Sep 09 16:40:48.851498 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.851503    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33c68b8-749c-4a69-96b6-cdc447af2192" containerName="payload-processing"
Sep 09 16:40:48.854294 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.854277    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4"
Sep 09 16:40:48.857185 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.857164    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxghs\""
Sep 09 16:40:48.857185 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.857179    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:40:48.870228 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.870206    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4"]
Sep 09 16:40:48.975274 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:48.975231    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d5aa39-f581-4360-a803-e40bc1a47416-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4\" (UID: \"91d5aa39-f581-4360-a803-e40bc1a47416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4"
Sep 09 16:40:49.075821 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:49.075776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d5aa39-f581-4360-a803-e40bc1a47416-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4\" (UID: \"91d5aa39-f581-4360-a803-e40bc1a47416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4"
Sep 09 16:40:49.076364 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:49.076343    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d5aa39-f581-4360-a803-e40bc1a47416-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4\" (UID: \"91d5aa39-f581-4360-a803-e40bc1a47416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4"
Sep 09 16:40:49.164714 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:49.164628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4"
Sep 09 16:40:49.433094 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:49.433071    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4"]
Sep 09 16:40:49.435913 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:40:49.435885    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d5aa39_f581_4360_a803_e40bc1a47416.slice/crio-2c0c3e33829f4f8eae543dd07f9d99b5cfe2281e08560e330eccc31b319b03bd WatchSource:0}: Error finding container 2c0c3e33829f4f8eae543dd07f9d99b5cfe2281e08560e330eccc31b319b03bd: Status 404 returned error can't find the container with id 2c0c3e33829f4f8eae543dd07f9d99b5cfe2281e08560e330eccc31b319b03bd
Sep 09 16:40:49.512967 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:49.512935    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4" event={"ID":"91d5aa39-f581-4360-a803-e40bc1a47416","Type":"ContainerStarted","Data":"2c0c3e33829f4f8eae543dd07f9d99b5cfe2281e08560e330eccc31b319b03bd"}
Sep 09 16:40:50.519769 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:50.519732    2576 generic.go:356] "Generic (PLEG): container finished" podID="91d5aa39-f581-4360-a803-e40bc1a47416" containerID="9ef00cfdde888090bd9e951b2c7cc76e4ba765ec2b838ea86369a827e6bc0dce" exitCode=0
Sep 09 16:40:50.520125 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:50.519805    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4" event={"ID":"91d5aa39-f581-4360-a803-e40bc1a47416","Type":"ContainerDied","Data":"9ef00cfdde888090bd9e951b2c7cc76e4ba765ec2b838ea86369a827e6bc0dce"}
Sep 09 16:40:51.647814 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:51.647791    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4"
Sep 09 16:40:51.699467 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:51.699433    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d5aa39-f581-4360-a803-e40bc1a47416-maas-api-service-ca\") pod \"91d5aa39-f581-4360-a803-e40bc1a47416\" (UID: \"91d5aa39-f581-4360-a803-e40bc1a47416\") "
Sep 09 16:40:51.699840 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:51.699818    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d5aa39-f581-4360-a803-e40bc1a47416-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91d5aa39-f581-4360-a803-e40bc1a47416" (UID: "91d5aa39-f581-4360-a803-e40bc1a47416"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:40:51.800578 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:51.800505    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d5aa39-f581-4360-a803-e40bc1a47416-maas-api-service-ca\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:52.530303 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:52.530273    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4"
Sep 09 16:40:52.530474 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:52.530273    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4" event={"ID":"91d5aa39-f581-4360-a803-e40bc1a47416","Type":"ContainerDied","Data":"2c0c3e33829f4f8eae543dd07f9d99b5cfe2281e08560e330eccc31b319b03bd"}
Sep 09 16:40:52.530474 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:52.530398    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c0c3e33829f4f8eae543dd07f9d99b5cfe2281e08560e330eccc31b319b03bd"
Sep 09 16:40:59.275951 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.275918    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-041bf1-29816190-fw5kl"]
Sep 09 16:40:59.282596 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.282561    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-041bf1-29816190-fw5kl"]
Sep 09 16:40:59.300293 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.300265    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"]
Sep 09 16:40:59.300538 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.300508    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7" podUID="eff2d442-febc-428f-92a4-1cd6690ce32a" containerName="payload-processing" containerID="cri-o://273e75f6d6618abd70598b0a2f88b978cc993c3ff38794bfe9855ca008811e8e" gracePeriod=30
Sep 09 16:40:59.308310 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.308224    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"]
Sep 09 16:40:59.308489 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.308439    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp" podUID="8d864c81-b058-493b-863a-3318edcdba59" containerName="payload-pre-processing" containerID="cri-o://911f5f5fb1d32637a9698856a620cce0a87bce6c1e3d62f1da33d8791d31dd6b" gracePeriod=30
Sep 09 16:40:59.467136 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.467099    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4"]
Sep 09 16:40:59.468487 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.468453    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-041bf1-x24c4"]
Sep 09 16:40:59.547674 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.547654    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"
Sep 09 16:40:59.563599 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.563553    2576 generic.go:356] "Generic (PLEG): container finished" podID="eff2d442-febc-428f-92a4-1cd6690ce32a" containerID="273e75f6d6618abd70598b0a2f88b978cc993c3ff38794bfe9855ca008811e8e" exitCode=0
Sep 09 16:40:59.563794 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.563597    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"
Sep 09 16:40:59.563794 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.563649    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7" event={"ID":"eff2d442-febc-428f-92a4-1cd6690ce32a","Type":"ContainerDied","Data":"273e75f6d6618abd70598b0a2f88b978cc993c3ff38794bfe9855ca008811e8e"}
Sep 09 16:40:59.563794 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.563713    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7" event={"ID":"eff2d442-febc-428f-92a4-1cd6690ce32a","Type":"ContainerDied","Data":"442649054e77082b1a8a5e0ca017e3c7ae3cd16b9c9c0c54447a4e81c95f4f17"}
Sep 09 16:40:59.563794 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.563745    2576 scope.go:117] "RemoveContainer" containerID="273e75f6d6618abd70598b0a2f88b978cc993c3ff38794bfe9855ca008811e8e"
Sep 09 16:40:59.566398 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.566371    2576 generic.go:356] "Generic (PLEG): container finished" podID="8d864c81-b058-493b-863a-3318edcdba59" containerID="911f5f5fb1d32637a9698856a620cce0a87bce6c1e3d62f1da33d8791d31dd6b" exitCode=0
Sep 09 16:40:59.566518 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.566475    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp" event={"ID":"8d864c81-b058-493b-863a-3318edcdba59","Type":"ContainerDied","Data":"911f5f5fb1d32637a9698856a620cce0a87bce6c1e3d62f1da33d8791d31dd6b"}
Sep 09 16:40:59.577506 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.577492    2576 scope.go:117] "RemoveContainer" containerID="273e75f6d6618abd70598b0a2f88b978cc993c3ff38794bfe9855ca008811e8e"
Sep 09 16:40:59.577768 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:40:59.577747    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273e75f6d6618abd70598b0a2f88b978cc993c3ff38794bfe9855ca008811e8e\": container with ID starting with 273e75f6d6618abd70598b0a2f88b978cc993c3ff38794bfe9855ca008811e8e not found: ID does not exist" containerID="273e75f6d6618abd70598b0a2f88b978cc993c3ff38794bfe9855ca008811e8e"
Sep 09 16:40:59.577841 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.577778    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273e75f6d6618abd70598b0a2f88b978cc993c3ff38794bfe9855ca008811e8e"} err="failed to get container status \"273e75f6d6618abd70598b0a2f88b978cc993c3ff38794bfe9855ca008811e8e\": rpc error: code = NotFound desc = could not find container \"273e75f6d6618abd70598b0a2f88b978cc993c3ff38794bfe9855ca008811e8e\": container with ID starting with 273e75f6d6618abd70598b0a2f88b978cc993c3ff38794bfe9855ca008811e8e not found: ID does not exist"
Sep 09 16:40:59.581295 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.581275    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"
Sep 09 16:40:59.674032 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.674000    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j79r\" (UniqueName: \"kubernetes.io/projected/8d864c81-b058-493b-863a-3318edcdba59-kube-api-access-8j79r\") pod \"8d864c81-b058-493b-863a-3318edcdba59\" (UID: \"8d864c81-b058-493b-863a-3318edcdba59\") "
Sep 09 16:40:59.674181 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.674052    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d864c81-b058-493b-863a-3318edcdba59-plugins-config-volume\") pod \"8d864c81-b058-493b-863a-3318edcdba59\" (UID: \"8d864c81-b058-493b-863a-3318edcdba59\") "
Sep 09 16:40:59.674223 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.674201    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff2d442-febc-428f-92a4-1cd6690ce32a-plugins-config-volume\") pod \"eff2d442-febc-428f-92a4-1cd6690ce32a\" (UID: \"eff2d442-febc-428f-92a4-1cd6690ce32a\") "
Sep 09 16:40:59.674280 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.674264    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w55rk\" (UniqueName: \"kubernetes.io/projected/eff2d442-febc-428f-92a4-1cd6690ce32a-kube-api-access-w55rk\") pod \"eff2d442-febc-428f-92a4-1cd6690ce32a\" (UID: \"eff2d442-febc-428f-92a4-1cd6690ce32a\") "
Sep 09 16:40:59.674529 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.674499    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d864c81-b058-493b-863a-3318edcdba59-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d864c81-b058-493b-863a-3318edcdba59" (UID: "8d864c81-b058-493b-863a-3318edcdba59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:40:59.674529 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.674511    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff2d442-febc-428f-92a4-1cd6690ce32a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eff2d442-febc-428f-92a4-1cd6690ce32a" (UID: "eff2d442-febc-428f-92a4-1cd6690ce32a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:40:59.676365 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.676346    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff2d442-febc-428f-92a4-1cd6690ce32a-kube-api-access-w55rk" (OuterVolumeSpecName: "kube-api-access-w55rk") pod "eff2d442-febc-428f-92a4-1cd6690ce32a" (UID: "eff2d442-febc-428f-92a4-1cd6690ce32a"). InnerVolumeSpecName "kube-api-access-w55rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:40:59.676365 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.676355    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d864c81-b058-493b-863a-3318edcdba59-kube-api-access-8j79r" (OuterVolumeSpecName: "kube-api-access-8j79r") pod "8d864c81-b058-493b-863a-3318edcdba59" (UID: "8d864c81-b058-493b-863a-3318edcdba59"). InnerVolumeSpecName "kube-api-access-8j79r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:40:59.774863 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.774839    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d864c81-b058-493b-863a-3318edcdba59-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:59.774863 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.774860    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff2d442-febc-428f-92a4-1cd6690ce32a-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:59.774998 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.774869    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w55rk\" (UniqueName: \"kubernetes.io/projected/eff2d442-febc-428f-92a4-1cd6690ce32a-kube-api-access-w55rk\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:59.774998 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.774878    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j79r\" (UniqueName: \"kubernetes.io/projected/8d864c81-b058-493b-863a-3318edcdba59-kube-api-access-8j79r\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:40:59.883285 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.883255    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"]
Sep 09 16:40:59.885592 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:40:59.885570    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-041bf1-74f7c796bc-d2fp7"]
Sep 09 16:41:00.552980 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:00.552945    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91d5aa39-f581-4360-a803-e40bc1a47416" path="/var/lib/kubelet/pods/91d5aa39-f581-4360-a803-e40bc1a47416/volumes"
Sep 09 16:41:00.553333 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:00.553253    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b58f933-3179-45a8-ac10-a53faa2690a4" path="/var/lib/kubelet/pods/9b58f933-3179-45a8-ac10-a53faa2690a4/volumes"
Sep 09 16:41:00.553530 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:00.553517    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff2d442-febc-428f-92a4-1cd6690ce32a" path="/var/lib/kubelet/pods/eff2d442-febc-428f-92a4-1cd6690ce32a/volumes"
Sep 09 16:41:00.573576 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:00.573511    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp" event={"ID":"8d864c81-b058-493b-863a-3318edcdba59","Type":"ContainerDied","Data":"9f889ce67b6af59fa96b6359a363c7a880808f5a339b789c6d47e8aefb68a092"}
Sep 09 16:41:00.573576 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:00.573537    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"
Sep 09 16:41:00.573576 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:00.573560    2576 scope.go:117] "RemoveContainer" containerID="911f5f5fb1d32637a9698856a620cce0a87bce6c1e3d62f1da33d8791d31dd6b"
Sep 09 16:41:00.590396 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:00.590374    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"]
Sep 09 16:41:00.594069 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:00.594048    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-041bf1-d487c4cb6-44npp"]
Sep 09 16:41:02.552685 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:02.552650    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d864c81-b058-493b-863a-3318edcdba59" path="/var/lib/kubelet/pods/8d864c81-b058-493b-863a-3318edcdba59/volumes"
Sep 09 16:41:06.672877 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.672843    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29"]
Sep 09 16:41:06.673341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.673137    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d864c81-b058-493b-863a-3318edcdba59" containerName="payload-pre-processing"
Sep 09 16:41:06.673341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.673148    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d864c81-b058-493b-863a-3318edcdba59" containerName="payload-pre-processing"
Sep 09 16:41:06.673341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.673155    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91d5aa39-f581-4360-a803-e40bc1a47416" containerName="revoke-keys"
Sep 09 16:41:06.673341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.673162    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="91d5aa39-f581-4360-a803-e40bc1a47416" containerName="revoke-keys"
Sep 09 16:41:06.673341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.673180    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff2d442-febc-428f-92a4-1cd6690ce32a" containerName="payload-processing"
Sep 09 16:41:06.673341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.673185    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff2d442-febc-428f-92a4-1cd6690ce32a" containerName="payload-processing"
Sep 09 16:41:06.673341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.673229    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff2d442-febc-428f-92a4-1cd6690ce32a" containerName="payload-processing"
Sep 09 16:41:06.673341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.673238    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="91d5aa39-f581-4360-a803-e40bc1a47416" containerName="revoke-keys"
Sep 09 16:41:06.673341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.673245    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d864c81-b058-493b-863a-3318edcdba59" containerName="payload-pre-processing"
Sep 09 16:41:06.677119 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.677100    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29"
Sep 09 16:41:06.679440 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.679415    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:41:06.679546 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.679419    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxghs\""
Sep 09 16:41:06.684781 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.684760    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29"]
Sep 09 16:41:06.832618 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.832582    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed66b76-9542-4e40-8629-7f3c50681699-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29\" (UID: \"3ed66b76-9542-4e40-8629-7f3c50681699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29"
Sep 09 16:41:06.933439 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.933352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed66b76-9542-4e40-8629-7f3c50681699-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29\" (UID: \"3ed66b76-9542-4e40-8629-7f3c50681699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29"
Sep 09 16:41:06.933980 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.933960    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed66b76-9542-4e40-8629-7f3c50681699-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29\" (UID: \"3ed66b76-9542-4e40-8629-7f3c50681699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29"
Sep 09 16:41:06.987438 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:06.987403    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29"
Sep 09 16:41:07.255229 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:07.255193    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29"]
Sep 09 16:41:07.258356 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:41:07.258333    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed66b76_9542_4e40_8629_7f3c50681699.slice/crio-0f3e2c1e8f1b0173f3c27bb333cae3dd91cef08db7c2f965cc368d31b6bb3ae5 WatchSource:0}: Error finding container 0f3e2c1e8f1b0173f3c27bb333cae3dd91cef08db7c2f965cc368d31b6bb3ae5: Status 404 returned error can't find the container with id 0f3e2c1e8f1b0173f3c27bb333cae3dd91cef08db7c2f965cc368d31b6bb3ae5
Sep 09 16:41:07.607022 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:07.606984    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29" event={"ID":"3ed66b76-9542-4e40-8629-7f3c50681699","Type":"ContainerStarted","Data":"3e1a377f193b6453f3742ab32090bb90ea5daa1605155a5533da549f8fdfc6ae"}
Sep 09 16:41:07.607200 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:07.607034    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29" event={"ID":"3ed66b76-9542-4e40-8629-7f3c50681699","Type":"ContainerStarted","Data":"0f3e2c1e8f1b0173f3c27bb333cae3dd91cef08db7c2f965cc368d31b6bb3ae5"}
Sep 09 16:41:07.622923 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:07.622879    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29" podStartSLOduration=1.6228641129999999 podStartE2EDuration="1.622864113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:41:07.62135931 +0000 UTC m=+1943.556977754" watchObservedRunningTime="2026-09-09 16:41:07.622864113 +0000 UTC m=+1943.558482529"
Sep 09 16:41:08.612851 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:08.612815    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ed66b76-9542-4e40-8629-7f3c50681699" containerID="3e1a377f193b6453f3742ab32090bb90ea5daa1605155a5533da549f8fdfc6ae" exitCode=0
Sep 09 16:41:08.613255 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:08.612903    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29" event={"ID":"3ed66b76-9542-4e40-8629-7f3c50681699","Type":"ContainerDied","Data":"3e1a377f193b6453f3742ab32090bb90ea5daa1605155a5533da549f8fdfc6ae"}
Sep 09 16:41:09.742381 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:09.742354    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29"
Sep 09 16:41:09.863387 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:09.863355    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed66b76-9542-4e40-8629-7f3c50681699-maas-api-service-ca\") pod \"3ed66b76-9542-4e40-8629-7f3c50681699\" (UID: \"3ed66b76-9542-4e40-8629-7f3c50681699\") "
Sep 09 16:41:09.863756 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:09.863733    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed66b76-9542-4e40-8629-7f3c50681699-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ed66b76-9542-4e40-8629-7f3c50681699" (UID: "3ed66b76-9542-4e40-8629-7f3c50681699"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:41:09.964394 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:09.964313    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed66b76-9542-4e40-8629-7f3c50681699-maas-api-service-ca\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:41:10.623175 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:10.623149    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29"
Sep 09 16:41:10.623375 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:10.623148    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29" event={"ID":"3ed66b76-9542-4e40-8629-7f3c50681699","Type":"ContainerDied","Data":"0f3e2c1e8f1b0173f3c27bb333cae3dd91cef08db7c2f965cc368d31b6bb3ae5"}
Sep 09 16:41:10.623375 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:10.623277    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f3e2c1e8f1b0173f3c27bb333cae3dd91cef08db7c2f965cc368d31b6bb3ae5"
Sep 09 16:41:17.257387 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.257357    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"]
Sep 09 16:41:17.257809 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.257565    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5" podUID="8ea89d17-f59c-413d-848f-0bf035f59da7" containerName="payload-processing" containerID="cri-o://21000a445eaac40e0350ac38beea33a01196287051f1d689680c578ed46155ee" gracePeriod=30
Sep 09 16:41:17.265559 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.265534    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"]
Sep 09 16:41:17.265781 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.265756    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm" podUID="6b4187cd-3eeb-47c4-b885-aa26f2b9d306" containerName="payload-pre-processing" containerID="cri-o://b23276beee4f0421fc4969d2a88f2efecc05ae7baea0ca65ed90a26f2a15061a" gracePeriod=30
Sep 09 16:41:17.412931 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.412853    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29"]
Sep 09 16:41:17.414550 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.414522    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9520a4-tzw29"]
Sep 09 16:41:17.541158 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.541138    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"
Sep 09 16:41:17.543787 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.543767    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"
Sep 09 16:41:17.653786 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.653752    2576 generic.go:356] "Generic (PLEG): container finished" podID="8ea89d17-f59c-413d-848f-0bf035f59da7" containerID="21000a445eaac40e0350ac38beea33a01196287051f1d689680c578ed46155ee" exitCode=0
Sep 09 16:41:17.653961 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.653790    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"
Sep 09 16:41:17.653961 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.653844    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5" event={"ID":"8ea89d17-f59c-413d-848f-0bf035f59da7","Type":"ContainerDied","Data":"21000a445eaac40e0350ac38beea33a01196287051f1d689680c578ed46155ee"}
Sep 09 16:41:17.653961 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.653887    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5" event={"ID":"8ea89d17-f59c-413d-848f-0bf035f59da7","Type":"ContainerDied","Data":"30d4d8725368f2ef2c1c2d1dbc019f70d2b20979659c2bc78bd9eccf3a31f0ec"}
Sep 09 16:41:17.653961 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.653909    2576 scope.go:117] "RemoveContainer" containerID="21000a445eaac40e0350ac38beea33a01196287051f1d689680c578ed46155ee"
Sep 09 16:41:17.655386 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.655297    2576 generic.go:356] "Generic (PLEG): container finished" podID="6b4187cd-3eeb-47c4-b885-aa26f2b9d306" containerID="b23276beee4f0421fc4969d2a88f2efecc05ae7baea0ca65ed90a26f2a15061a" exitCode=0
Sep 09 16:41:17.655386 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.655329    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm" event={"ID":"6b4187cd-3eeb-47c4-b885-aa26f2b9d306","Type":"ContainerDied","Data":"b23276beee4f0421fc4969d2a88f2efecc05ae7baea0ca65ed90a26f2a15061a"}
Sep 09 16:41:17.655386 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.655343    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"
Sep 09 16:41:17.655386 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.655360    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm" event={"ID":"6b4187cd-3eeb-47c4-b885-aa26f2b9d306","Type":"ContainerDied","Data":"8688bb43fd820d883be0584c277aa22c501c48f21dfdb076ca8956b239131340"}
Sep 09 16:41:17.662139 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.662116    2576 scope.go:117] "RemoveContainer" containerID="21000a445eaac40e0350ac38beea33a01196287051f1d689680c578ed46155ee"
Sep 09 16:41:17.662426 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:41:17.662405    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21000a445eaac40e0350ac38beea33a01196287051f1d689680c578ed46155ee\": container with ID starting with 21000a445eaac40e0350ac38beea33a01196287051f1d689680c578ed46155ee not found: ID does not exist" containerID="21000a445eaac40e0350ac38beea33a01196287051f1d689680c578ed46155ee"
Sep 09 16:41:17.662483 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.662434    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21000a445eaac40e0350ac38beea33a01196287051f1d689680c578ed46155ee"} err="failed to get container status \"21000a445eaac40e0350ac38beea33a01196287051f1d689680c578ed46155ee\": rpc error: code = NotFound desc = could not find container \"21000a445eaac40e0350ac38beea33a01196287051f1d689680c578ed46155ee\": container with ID starting with 21000a445eaac40e0350ac38beea33a01196287051f1d689680c578ed46155ee not found: ID does not exist"
Sep 09 16:41:17.662483 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.662450    2576 scope.go:117] "RemoveContainer" containerID="b23276beee4f0421fc4969d2a88f2efecc05ae7baea0ca65ed90a26f2a15061a"
Sep 09 16:41:17.669791 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.669774    2576 scope.go:117] "RemoveContainer" containerID="b23276beee4f0421fc4969d2a88f2efecc05ae7baea0ca65ed90a26f2a15061a"
Sep 09 16:41:17.670036 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:41:17.670017    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b23276beee4f0421fc4969d2a88f2efecc05ae7baea0ca65ed90a26f2a15061a\": container with ID starting with b23276beee4f0421fc4969d2a88f2efecc05ae7baea0ca65ed90a26f2a15061a not found: ID does not exist" containerID="b23276beee4f0421fc4969d2a88f2efecc05ae7baea0ca65ed90a26f2a15061a"
Sep 09 16:41:17.670084 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.670042    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b23276beee4f0421fc4969d2a88f2efecc05ae7baea0ca65ed90a26f2a15061a"} err="failed to get container status \"b23276beee4f0421fc4969d2a88f2efecc05ae7baea0ca65ed90a26f2a15061a\": rpc error: code = NotFound desc = could not find container \"b23276beee4f0421fc4969d2a88f2efecc05ae7baea0ca65ed90a26f2a15061a\": container with ID starting with b23276beee4f0421fc4969d2a88f2efecc05ae7baea0ca65ed90a26f2a15061a not found: ID does not exist"
Sep 09 16:41:17.730493 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.730470    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbmkz\" (UniqueName: \"kubernetes.io/projected/8ea89d17-f59c-413d-848f-0bf035f59da7-kube-api-access-hbmkz\") pod \"8ea89d17-f59c-413d-848f-0bf035f59da7\" (UID: \"8ea89d17-f59c-413d-848f-0bf035f59da7\") "
Sep 09 16:41:17.730593 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.730532    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4187cd-3eeb-47c4-b885-aa26f2b9d306-plugins-config-volume\") pod \"6b4187cd-3eeb-47c4-b885-aa26f2b9d306\" (UID: \"6b4187cd-3eeb-47c4-b885-aa26f2b9d306\") "
Sep 09 16:41:17.730593 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.730588    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea89d17-f59c-413d-848f-0bf035f59da7-plugins-config-volume\") pod \"8ea89d17-f59c-413d-848f-0bf035f59da7\" (UID: \"8ea89d17-f59c-413d-848f-0bf035f59da7\") "
Sep 09 16:41:17.730740 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.730637    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2kkv\" (UniqueName: \"kubernetes.io/projected/6b4187cd-3eeb-47c4-b885-aa26f2b9d306-kube-api-access-r2kkv\") pod \"6b4187cd-3eeb-47c4-b885-aa26f2b9d306\" (UID: \"6b4187cd-3eeb-47c4-b885-aa26f2b9d306\") "
Sep 09 16:41:17.730904 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.730880    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4187cd-3eeb-47c4-b885-aa26f2b9d306-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b4187cd-3eeb-47c4-b885-aa26f2b9d306" (UID: "6b4187cd-3eeb-47c4-b885-aa26f2b9d306"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:41:17.730999 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.730952    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea89d17-f59c-413d-848f-0bf035f59da7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ea89d17-f59c-413d-848f-0bf035f59da7" (UID: "8ea89d17-f59c-413d-848f-0bf035f59da7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:41:17.732789 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.732766    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4187cd-3eeb-47c4-b885-aa26f2b9d306-kube-api-access-r2kkv" (OuterVolumeSpecName: "kube-api-access-r2kkv") pod "6b4187cd-3eeb-47c4-b885-aa26f2b9d306" (UID: "6b4187cd-3eeb-47c4-b885-aa26f2b9d306"). InnerVolumeSpecName "kube-api-access-r2kkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:41:17.732920 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.732905    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea89d17-f59c-413d-848f-0bf035f59da7-kube-api-access-hbmkz" (OuterVolumeSpecName: "kube-api-access-hbmkz") pod "8ea89d17-f59c-413d-848f-0bf035f59da7" (UID: "8ea89d17-f59c-413d-848f-0bf035f59da7"). InnerVolumeSpecName "kube-api-access-hbmkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:41:17.831941 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.831867    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea89d17-f59c-413d-848f-0bf035f59da7-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:41:17.831941 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.831894    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2kkv\" (UniqueName: \"kubernetes.io/projected/6b4187cd-3eeb-47c4-b885-aa26f2b9d306-kube-api-access-r2kkv\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:41:17.831941 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.831906    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbmkz\" (UniqueName: \"kubernetes.io/projected/8ea89d17-f59c-413d-848f-0bf035f59da7-kube-api-access-hbmkz\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:41:17.831941 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.831921    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4187cd-3eeb-47c4-b885-aa26f2b9d306-plugins-config-volume\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:41:17.974117 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.974092    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"]
Sep 09 16:41:17.977664 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.977642    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9520a4-b76694cb7-xm7p5"]
Sep 09 16:41:17.986404 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.986382    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"]
Sep 09 16:41:17.988353 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:17.988322    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9520a4-bf66c4bcd-c2snm"]
Sep 09 16:41:18.552870 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:18.552834    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ed66b76-9542-4e40-8629-7f3c50681699" path="/var/lib/kubelet/pods/3ed66b76-9542-4e40-8629-7f3c50681699/volumes"
Sep 09 16:41:18.553238 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:18.553185    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b4187cd-3eeb-47c4-b885-aa26f2b9d306" path="/var/lib/kubelet/pods/6b4187cd-3eeb-47c4-b885-aa26f2b9d306/volumes"
Sep 09 16:41:18.553488 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:18.553476    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ea89d17-f59c-413d-848f-0bf035f59da7" path="/var/lib/kubelet/pods/8ea89d17-f59c-413d-848f-0bf035f59da7/volumes"
Sep 09 16:41:44.834782 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:44.834683    2576 scope.go:117] "RemoveContainer" containerID="d00bf8fbf9ef4a2922d08cb762392b290fa34b9c15efd7db33d26925ab61dd40"
Sep 09 16:41:44.843059 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:41:44.843040    2576 scope.go:117] "RemoveContainer" containerID="234841c7301c2d734bef0050663d5331ed3e2609d6ee009648ccdd8939e094f9"
Sep 09 16:42:44.876800 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:42:44.876763    2576 scope.go:117] "RemoveContainer" containerID="28453f323bc093b7bbef7963716d5457bc126ebbf6c93ae549a84165fb4cb1a4"
Sep 09 16:42:44.884964 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:42:44.884930    2576 scope.go:117] "RemoveContainer" containerID="a1ada3f591a4e3b6d68d7e96d623ba7544022af33669266041f6e92b364894f0"
Sep 09 16:42:44.894430 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:42:44.894397    2576 scope.go:117] "RemoveContainer" containerID="13fc88215e4754eafb33fce907f7fadf9b149d0377cb6a33f1b292ee098540ba"
Sep 09 16:45:00.140567 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.140488    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816205-dvwsh"]
Sep 09 16:45:00.142887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.140858    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed66b76-9542-4e40-8629-7f3c50681699" containerName="revoke-keys"
Sep 09 16:45:00.142887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.140873    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed66b76-9542-4e40-8629-7f3c50681699" containerName="revoke-keys"
Sep 09 16:45:00.142887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.140883    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b4187cd-3eeb-47c4-b885-aa26f2b9d306" containerName="payload-pre-processing"
Sep 09 16:45:00.142887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.140888    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b4187cd-3eeb-47c4-b885-aa26f2b9d306" containerName="payload-pre-processing"
Sep 09 16:45:00.142887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.140897    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea89d17-f59c-413d-848f-0bf035f59da7" containerName="payload-processing"
Sep 09 16:45:00.142887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.140903    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea89d17-f59c-413d-848f-0bf035f59da7" containerName="payload-processing"
Sep 09 16:45:00.142887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.140951    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ea89d17-f59c-413d-848f-0bf035f59da7" containerName="payload-processing"
Sep 09 16:45:00.142887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.140958    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed66b76-9542-4e40-8629-7f3c50681699" containerName="revoke-keys"
Sep 09 16:45:00.142887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.140965    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b4187cd-3eeb-47c4-b885-aa26f2b9d306" containerName="payload-pre-processing"
Sep 09 16:45:00.143815 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.143800    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816205-dvwsh"
Sep 09 16:45:00.146309 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.146288    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxghs\""
Sep 09 16:45:00.151001 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.150979    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816205-dvwsh"]
Sep 09 16:45:00.153809 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.153788    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816205-dvwsh"
Sep 09 16:45:00.424411 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.424236    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816205-dvwsh"]
Sep 09 16:45:00.426974 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:45:00.426946    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63b2864_5684_4762_a838_f02452553acf.slice/crio-3d77383954417cd70a5fdb25cd86fa30afd7ed9c107600a5f0393221a81382f9 WatchSource:0}: Error finding container 3d77383954417cd70a5fdb25cd86fa30afd7ed9c107600a5f0393221a81382f9: Status 404 returned error can't find the container with id 3d77383954417cd70a5fdb25cd86fa30afd7ed9c107600a5f0393221a81382f9
Sep 09 16:45:00.611136 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:00.611102    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816205-dvwsh" event={"ID":"b63b2864-5684-4762-a838-f02452553acf","Type":"ContainerStarted","Data":"3d77383954417cd70a5fdb25cd86fa30afd7ed9c107600a5f0393221a81382f9"}
Sep 09 16:45:01.617105 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:01.617014    2576 generic.go:356] "Generic (PLEG): container finished" podID="b63b2864-5684-4762-a838-f02452553acf" containerID="0a548bb06442a7f2e6af8ba0d05042420ebc62375b344f74af3aad4675c4cd41" exitCode=0
Sep 09 16:45:01.617461 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:01.617102    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816205-dvwsh" event={"ID":"b63b2864-5684-4762-a838-f02452553acf","Type":"ContainerDied","Data":"0a548bb06442a7f2e6af8ba0d05042420ebc62375b344f74af3aad4675c4cd41"}
Sep 09 16:45:02.754090 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:02.754060    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816205-dvwsh"
Sep 09 16:45:03.627066 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:03.627040    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816205-dvwsh"
Sep 09 16:45:03.627256 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:03.627039    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816205-dvwsh" event={"ID":"b63b2864-5684-4762-a838-f02452553acf","Type":"ContainerDied","Data":"3d77383954417cd70a5fdb25cd86fa30afd7ed9c107600a5f0393221a81382f9"}
Sep 09 16:45:03.627256 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:03.627160    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d77383954417cd70a5fdb25cd86fa30afd7ed9c107600a5f0393221a81382f9"
Sep 09 16:45:44.929268 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:44.929237    2576 scope.go:117] "RemoveContainer" containerID="b897931c6cb6c83834600c90f79aef7e9ea6c967a6decc2580f42a3ac0ee25a9"
Sep 09 16:45:44.937111 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:44.937091    2576 scope.go:117] "RemoveContainer" containerID="af25a25597510754815845c528c605cf8da2f085a651f8c9377b6ef64e0e2102"
Sep 09 16:45:57.414396 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.414362    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"]
Sep 09 16:45:57.414781 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.414669    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b63b2864-5684-4762-a838-f02452553acf" containerName="cleanup"
Sep 09 16:45:57.414781 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.414682    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b63b2864-5684-4762-a838-f02452553acf" containerName="cleanup"
Sep 09 16:45:57.414781 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.414758    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b63b2864-5684-4762-a838-f02452553acf" containerName="cleanup"
Sep 09 16:45:57.417632 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.417610    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.419854 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.419834    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-8405f9cc-kserve-self-signed-certs\""
Sep 09 16:45:57.432376 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.432352    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"]
Sep 09 16:45:57.539604 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.539572    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-dshm\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.539604 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.539608    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-home\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.539836 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.539628    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-model-cache\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.539836 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.539719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvctj\" (UniqueName: \"kubernetes.io/projected/a829b1d5-2d38-44e7-ab20-a14e402bc37e-kube-api-access-nvctj\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.539836 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.539773    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-tmp-dir\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.539836 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.539814    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a829b1d5-2d38-44e7-ab20-a14e402bc37e-tls-certs\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.641065 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.641022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvctj\" (UniqueName: \"kubernetes.io/projected/a829b1d5-2d38-44e7-ab20-a14e402bc37e-kube-api-access-nvctj\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.641065 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.641071    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-tmp-dir\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.641295 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.641110    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a829b1d5-2d38-44e7-ab20-a14e402bc37e-tls-certs\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.641295 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.641149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-dshm\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.641295 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.641178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-home\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.641295 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.641204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-model-cache\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.641565 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.641543    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-tmp-dir\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.641635 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.641563    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-model-cache\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.641635 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.641606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-home\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.643707 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.643666    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-dshm\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.643878 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.643863    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a829b1d5-2d38-44e7-ab20-a14e402bc37e-tls-certs\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.648968 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.648949    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvctj\" (UniqueName: \"kubernetes.io/projected/a829b1d5-2d38-44e7-ab20-a14e402bc37e-kube-api-access-nvctj\") pod \"e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") " pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.726617 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.726582    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:45:57.997081 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:57.997051    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"]
Sep 09 16:45:57.999816 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:45:57.999789    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda829b1d5_2d38_44e7_ab20_a14e402bc37e.slice/crio-abea0d9bce0bec97c3912608f3e5c532007463e3517a8e901f903c4b94572baa WatchSource:0}: Error finding container abea0d9bce0bec97c3912608f3e5c532007463e3517a8e901f903c4b94572baa: Status 404 returned error can't find the container with id abea0d9bce0bec97c3912608f3e5c532007463e3517a8e901f903c4b94572baa
Sep 09 16:45:58.855764 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:58.855724    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd" event={"ID":"a829b1d5-2d38-44e7-ab20-a14e402bc37e","Type":"ContainerStarted","Data":"b63059b5887a5a37cc3ba04ee4dff8a42cf295ce592d9a207786b9de2bd19aa7"}
Sep 09 16:45:58.855764 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:58.855768    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd" event={"ID":"a829b1d5-2d38-44e7-ab20-a14e402bc37e","Type":"ContainerStarted","Data":"abea0d9bce0bec97c3912608f3e5c532007463e3517a8e901f903c4b94572baa"}
Sep 09 16:45:58.872807 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:45:58.872759    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd" podStartSLOduration=1.872743182 podStartE2EDuration="1.872743182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:45:58.87214695 +0000 UTC m=+2234.807765380" watchObservedRunningTime="2026-09-09 16:45:58.872743182 +0000 UTC m=+2234.808361648"
Sep 09 16:46:04.418279 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.418246    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"]
Sep 09 16:46:04.421354 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.421339    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.424341 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.424318    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-8405f9cc-kserve-self-signed-certs\""
Sep 09 16:46:04.431105 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.431082    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"]
Sep 09 16:46:04.494845 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.494817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-model-cache\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.494964 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.494868    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-tmp-dir\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.494964 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.494883    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56df47ff-dbdd-4dd5-b6db-465f2a73a346-tls-certs\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.494964 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.494931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-home\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.495104 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.494967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddm7h\" (UniqueName: \"kubernetes.io/projected/56df47ff-dbdd-4dd5-b6db-465f2a73a346-kube-api-access-ddm7h\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.495104 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.495001    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-dshm\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.595991 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.595961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-dshm\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.596164 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.596067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-model-cache\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.596272 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.596250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-tmp-dir\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.596410 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.596388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56df47ff-dbdd-4dd5-b6db-465f2a73a346-tls-certs\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.596552 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.596428    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-home\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.596552 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.596472    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddm7h\" (UniqueName: \"kubernetes.io/projected/56df47ff-dbdd-4dd5-b6db-465f2a73a346-kube-api-access-ddm7h\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.596552 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.596486    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-model-cache\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.596763 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.596663    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-tmp-dir\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.596820 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.596786    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-home\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.598477 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.598458    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-dshm\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.598889 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.598871    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56df47ff-dbdd-4dd5-b6db-465f2a73a346-tls-certs\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.604334 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.604316    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddm7h\" (UniqueName: \"kubernetes.io/projected/56df47ff-dbdd-4dd5-b6db-465f2a73a346-kube-api-access-ddm7h\") pod \"e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") " pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:04.730927 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:04.730896    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:05.000641 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:05.000560    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"]
Sep 09 16:46:05.003515 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:46:05.003489    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56df47ff_dbdd_4dd5_b6db_465f2a73a346.slice/crio-21bea635e61be8f982662709c82f79ff55e9d3fff7bc72ba4027804494a91810 WatchSource:0}: Error finding container 21bea635e61be8f982662709c82f79ff55e9d3fff7bc72ba4027804494a91810: Status 404 returned error can't find the container with id 21bea635e61be8f982662709c82f79ff55e9d3fff7bc72ba4027804494a91810
Sep 09 16:46:05.892330 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:05.892291    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s" event={"ID":"56df47ff-dbdd-4dd5-b6db-465f2a73a346","Type":"ContainerStarted","Data":"cdeeaf2f7450289dd0064095c4157170e73a5286306813e3d4298bf01dc5c023"}
Sep 09 16:46:05.892330 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:05.892335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s" event={"ID":"56df47ff-dbdd-4dd5-b6db-465f2a73a346","Type":"ContainerStarted","Data":"21bea635e61be8f982662709c82f79ff55e9d3fff7bc72ba4027804494a91810"}
Sep 09 16:46:05.909683 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:05.909630    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s" podStartSLOduration=1.9096160370000002 podStartE2EDuration="1.909616037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:46:05.908389246 +0000 UTC m=+2241.844007658" watchObservedRunningTime="2026-09-09 16:46:05.909616037 +0000 UTC m=+2241.845234452"
Sep 09 16:46:07.726674 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:07.726642    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:46:07.726674 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:07.726687    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:46:07.739610 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:07.739587    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:46:07.912603 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:07.912574    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:46:14.731967 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:14.731929    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:14.732359 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:14.731990    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:14.744469 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:14.744440    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:14.943281 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:14.943247    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:26.500797 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:26.500758    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"]
Sep 09 16:46:28.510334 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:28.510257    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"]
Sep 09 16:46:41.520003 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.519933    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd" podUID="a829b1d5-2d38-44e7-ab20-a14e402bc37e" containerName="main" containerID="cri-o://b63059b5887a5a37cc3ba04ee4dff8a42cf295ce592d9a207786b9de2bd19aa7" gracePeriod=45
Sep 09 16:46:41.762284 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.762261    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:46:41.800740 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.800659    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a829b1d5-2d38-44e7-ab20-a14e402bc37e-tls-certs\") pod \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") "
Sep 09 16:46:41.800865 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.800748    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-model-cache\") pod \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") "
Sep 09 16:46:41.800865 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.800784    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-dshm\") pod \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") "
Sep 09 16:46:41.800865 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.800853    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvctj\" (UniqueName: \"kubernetes.io/projected/a829b1d5-2d38-44e7-ab20-a14e402bc37e-kube-api-access-nvctj\") pod \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") "
Sep 09 16:46:41.801025 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.800877    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-tmp-dir\") pod \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") "
Sep 09 16:46:41.801025 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.800933    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-home\") pod \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\" (UID: \"a829b1d5-2d38-44e7-ab20-a14e402bc37e\") "
Sep 09 16:46:41.801148 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.801054    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-model-cache" (OuterVolumeSpecName: "model-cache") pod "a829b1d5-2d38-44e7-ab20-a14e402bc37e" (UID: "a829b1d5-2d38-44e7-ab20-a14e402bc37e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:46:41.801266 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.801240    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:46:41.801350 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.801281    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a829b1d5-2d38-44e7-ab20-a14e402bc37e" (UID: "a829b1d5-2d38-44e7-ab20-a14e402bc37e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:46:41.801350 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.801296    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-home" (OuterVolumeSpecName: "home") pod "a829b1d5-2d38-44e7-ab20-a14e402bc37e" (UID: "a829b1d5-2d38-44e7-ab20-a14e402bc37e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:46:41.803003 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.802982    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a829b1d5-2d38-44e7-ab20-a14e402bc37e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a829b1d5-2d38-44e7-ab20-a14e402bc37e" (UID: "a829b1d5-2d38-44e7-ab20-a14e402bc37e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:46:41.803100 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.803046    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a829b1d5-2d38-44e7-ab20-a14e402bc37e-kube-api-access-nvctj" (OuterVolumeSpecName: "kube-api-access-nvctj") pod "a829b1d5-2d38-44e7-ab20-a14e402bc37e" (UID: "a829b1d5-2d38-44e7-ab20-a14e402bc37e"). InnerVolumeSpecName "kube-api-access-nvctj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:46:41.803142 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.803116    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-dshm" (OuterVolumeSpecName: "dshm") pod "a829b1d5-2d38-44e7-ab20-a14e402bc37e" (UID: "a829b1d5-2d38-44e7-ab20-a14e402bc37e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:46:41.901954 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.901922    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:46:41.901954 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.901950    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a829b1d5-2d38-44e7-ab20-a14e402bc37e-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:46:41.902119 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.901963    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:46:41.902119 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.901975    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvctj\" (UniqueName: \"kubernetes.io/projected/a829b1d5-2d38-44e7-ab20-a14e402bc37e-kube-api-access-nvctj\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:46:41.902119 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:41.901987    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a829b1d5-2d38-44e7-ab20-a14e402bc37e-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:46:42.044825 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:42.044792    2576 generic.go:356] "Generic (PLEG): container finished" podID="a829b1d5-2d38-44e7-ab20-a14e402bc37e" containerID="b63059b5887a5a37cc3ba04ee4dff8a42cf295ce592d9a207786b9de2bd19aa7" exitCode=0
Sep 09 16:46:42.044972 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:42.044831    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"
Sep 09 16:46:42.044972 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:42.044839    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd" event={"ID":"a829b1d5-2d38-44e7-ab20-a14e402bc37e","Type":"ContainerDied","Data":"b63059b5887a5a37cc3ba04ee4dff8a42cf295ce592d9a207786b9de2bd19aa7"}
Sep 09 16:46:42.044972 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:42.044871    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd" event={"ID":"a829b1d5-2d38-44e7-ab20-a14e402bc37e","Type":"ContainerDied","Data":"abea0d9bce0bec97c3912608f3e5c532007463e3517a8e901f903c4b94572baa"}
Sep 09 16:46:42.044972 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:42.044888    2576 scope.go:117] "RemoveContainer" containerID="b63059b5887a5a37cc3ba04ee4dff8a42cf295ce592d9a207786b9de2bd19aa7"
Sep 09 16:46:42.053940 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:42.053923    2576 scope.go:117] "RemoveContainer" containerID="b63059b5887a5a37cc3ba04ee4dff8a42cf295ce592d9a207786b9de2bd19aa7"
Sep 09 16:46:42.054201 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:46:42.054184    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63059b5887a5a37cc3ba04ee4dff8a42cf295ce592d9a207786b9de2bd19aa7\": container with ID starting with b63059b5887a5a37cc3ba04ee4dff8a42cf295ce592d9a207786b9de2bd19aa7 not found: ID does not exist" containerID="b63059b5887a5a37cc3ba04ee4dff8a42cf295ce592d9a207786b9de2bd19aa7"
Sep 09 16:46:42.054247 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:42.054212    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63059b5887a5a37cc3ba04ee4dff8a42cf295ce592d9a207786b9de2bd19aa7"} err="failed to get container status \"b63059b5887a5a37cc3ba04ee4dff8a42cf295ce592d9a207786b9de2bd19aa7\": rpc error: code = NotFound desc = could not find container \"b63059b5887a5a37cc3ba04ee4dff8a42cf295ce592d9a207786b9de2bd19aa7\": container with ID starting with b63059b5887a5a37cc3ba04ee4dff8a42cf295ce592d9a207786b9de2bd19aa7 not found: ID does not exist"
Sep 09 16:46:42.065533 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:42.065507    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"]
Sep 09 16:46:42.068995 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:42.068974    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-8405f9cc-kserve-565fbd6bf9-skjpd"]
Sep 09 16:46:42.552727 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:42.552678    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a829b1d5-2d38-44e7-ab20-a14e402bc37e" path="/var/lib/kubelet/pods/a829b1d5-2d38-44e7-ab20-a14e402bc37e/volumes"
Sep 09 16:46:43.530589 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.530544    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s" podUID="56df47ff-dbdd-4dd5-b6db-465f2a73a346" containerName="main" containerID="cri-o://cdeeaf2f7450289dd0064095c4157170e73a5286306813e3d4298bf01dc5c023" gracePeriod=45
Sep 09 16:46:43.782887 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.782833    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:43.816655 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.816628    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-model-cache\") pod \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") "
Sep 09 16:46:43.816783 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.816660    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-home\") pod \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") "
Sep 09 16:46:43.816783 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.816689    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-dshm\") pod \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") "
Sep 09 16:46:43.816783 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.816751    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddm7h\" (UniqueName: \"kubernetes.io/projected/56df47ff-dbdd-4dd5-b6db-465f2a73a346-kube-api-access-ddm7h\") pod \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") "
Sep 09 16:46:43.816942 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.816796    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-tmp-dir\") pod \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") "
Sep 09 16:46:43.816942 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.816815    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56df47ff-dbdd-4dd5-b6db-465f2a73a346-tls-certs\") pod \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\" (UID: \"56df47ff-dbdd-4dd5-b6db-465f2a73a346\") "
Sep 09 16:46:43.817059 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.816969    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-model-cache" (OuterVolumeSpecName: "model-cache") pod "56df47ff-dbdd-4dd5-b6db-465f2a73a346" (UID: "56df47ff-dbdd-4dd5-b6db-465f2a73a346"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:46:43.817059 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.816979    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-home" (OuterVolumeSpecName: "home") pod "56df47ff-dbdd-4dd5-b6db-465f2a73a346" (UID: "56df47ff-dbdd-4dd5-b6db-465f2a73a346"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:46:43.817171 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.817149    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56df47ff-dbdd-4dd5-b6db-465f2a73a346" (UID: "56df47ff-dbdd-4dd5-b6db-465f2a73a346"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:46:43.819060 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.819033    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56df47ff-dbdd-4dd5-b6db-465f2a73a346-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56df47ff-dbdd-4dd5-b6db-465f2a73a346" (UID: "56df47ff-dbdd-4dd5-b6db-465f2a73a346"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:46:43.819338 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.819308    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-dshm" (OuterVolumeSpecName: "dshm") pod "56df47ff-dbdd-4dd5-b6db-465f2a73a346" (UID: "56df47ff-dbdd-4dd5-b6db-465f2a73a346"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:46:43.819432 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.819362    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56df47ff-dbdd-4dd5-b6db-465f2a73a346-kube-api-access-ddm7h" (OuterVolumeSpecName: "kube-api-access-ddm7h") pod "56df47ff-dbdd-4dd5-b6db-465f2a73a346" (UID: "56df47ff-dbdd-4dd5-b6db-465f2a73a346"). InnerVolumeSpecName "kube-api-access-ddm7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:46:43.918296 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.918265    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-tmp-dir\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:46:43.918296 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.918288    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56df47ff-dbdd-4dd5-b6db-465f2a73a346-tls-certs\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:46:43.918296 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.918298    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-model-cache\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:46:43.918494 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.918306    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-home\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:46:43.918494 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.918316    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56df47ff-dbdd-4dd5-b6db-465f2a73a346-dshm\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:46:43.918494 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:43.918325    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddm7h\" (UniqueName: \"kubernetes.io/projected/56df47ff-dbdd-4dd5-b6db-465f2a73a346-kube-api-access-ddm7h\") on node \"ip-10-0-137-117.ec2.internal\" DevicePath \"\""
Sep 09 16:46:44.057254 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:44.057170    2576 generic.go:356] "Generic (PLEG): container finished" podID="56df47ff-dbdd-4dd5-b6db-465f2a73a346" containerID="cdeeaf2f7450289dd0064095c4157170e73a5286306813e3d4298bf01dc5c023" exitCode=0
Sep 09 16:46:44.057254 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:44.057225    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"
Sep 09 16:46:44.057428 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:44.057231    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s" event={"ID":"56df47ff-dbdd-4dd5-b6db-465f2a73a346","Type":"ContainerDied","Data":"cdeeaf2f7450289dd0064095c4157170e73a5286306813e3d4298bf01dc5c023"}
Sep 09 16:46:44.057428 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:44.057355    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s" event={"ID":"56df47ff-dbdd-4dd5-b6db-465f2a73a346","Type":"ContainerDied","Data":"21bea635e61be8f982662709c82f79ff55e9d3fff7bc72ba4027804494a91810"}
Sep 09 16:46:44.057428 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:44.057385    2576 scope.go:117] "RemoveContainer" containerID="cdeeaf2f7450289dd0064095c4157170e73a5286306813e3d4298bf01dc5c023"
Sep 09 16:46:44.066021 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:44.066002    2576 scope.go:117] "RemoveContainer" containerID="cdeeaf2f7450289dd0064095c4157170e73a5286306813e3d4298bf01dc5c023"
Sep 09 16:46:44.066274 ip-10-0-137-117 kubenswrapper[2576]: E0909 16:46:44.066257    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdeeaf2f7450289dd0064095c4157170e73a5286306813e3d4298bf01dc5c023\": container with ID starting with cdeeaf2f7450289dd0064095c4157170e73a5286306813e3d4298bf01dc5c023 not found: ID does not exist" containerID="cdeeaf2f7450289dd0064095c4157170e73a5286306813e3d4298bf01dc5c023"
Sep 09 16:46:44.066319 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:44.066282    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdeeaf2f7450289dd0064095c4157170e73a5286306813e3d4298bf01dc5c023"} err="failed to get container status \"cdeeaf2f7450289dd0064095c4157170e73a5286306813e3d4298bf01dc5c023\": rpc error: code = NotFound desc = could not find container \"cdeeaf2f7450289dd0064095c4157170e73a5286306813e3d4298bf01dc5c023\": container with ID starting with cdeeaf2f7450289dd0064095c4157170e73a5286306813e3d4298bf01dc5c023 not found: ID does not exist"
Sep 09 16:46:44.076888 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:44.076868    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"]
Sep 09 16:46:44.082642 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:44.082622    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-8405f9cc-kserve-67bbf4847-s989s"]
Sep 09 16:46:44.555751 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:44.555719    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56df47ff-dbdd-4dd5-b6db-465f2a73a346" path="/var/lib/kubelet/pods/56df47ff-dbdd-4dd5-b6db-465f2a73a346/volumes"
Sep 09 16:46:44.957250 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:46:44.957176    2576 scope.go:117] "RemoveContainer" containerID="9cc3e3971297e0a80d73f47d15f78f7d5b44861341f7ffc36eba9a424320bbb0"
Sep 09 16:47:14.990420 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:14.990390    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f6f59dbd4-m82tr_41cb6b67-baf5-4685-a796-f4c08b233498/authorino/0.log"
Sep 09 16:47:21.450470 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:21.450440    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67c7bcf47f-dflt7_17c62f0e-b8f0-4669-bce9-c37adcd58a36/copy-manifests/0.log"
Sep 09 16:47:21.462781 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:21.462754    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67c7bcf47f-dflt7_17c62f0e-b8f0-4669-bce9-c37adcd58a36/manager/0.log"
Sep 09 16:47:22.345657 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:22.345628    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816205-dvwsh_b63b2864-5684-4762-a838-f02452553acf/cleanup/0.log"
Sep 09 16:47:23.312030 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:23.312000    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f6f59dbd4-m82tr_41cb6b67-baf5-4685-a796-f4c08b233498/authorino/0.log"
Sep 09 16:47:23.695004 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:23.694929    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-rz6hx_5afd1b8d-a975-4ea0-84da-d54d43a5e78b/kuadrant-console-plugin/0.log"
Sep 09 16:47:24.717574 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:24.717543    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54fc5d6dc5-4qr8n_b2893fe7-94e2-4115-8362-925f3908b0a6/kube-auth-proxy/0.log"
Sep 09 16:47:25.090626 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:25.090595    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-bjd4z_24d34adc-3182-4feb-acba-41b98f7c6a8b/payload-pre-processing/0.log"
Sep 09 16:47:25.217587 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:25.217555    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-s76z5_6bcd5da5-f6c4-46aa-87dd-21844af21310/payload-processing/0.log"
Sep 09 16:47:26.061971 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:26.061940    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-gnnxn_5acf5f60-eccf-48c5-9002-eb668daa5a4b/main/0.log"
Sep 09 16:47:26.184182 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:26.184154    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ft58lc_c88da762-3854-4b2b-b628-4297e4b1de46/main/0.log"
Sep 09 16:47:26.432417 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:26.432328    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-5lrr6_02ab45f4-89d6-47b1-9b4d-99184d95b633/main/0.log"
Sep 09 16:47:37.578763 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:37.578671    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-psbxz_27913e87-038f-437f-a49e-6e3768968886/global-pull-secret-syncer/0.log"
Sep 09 16:47:37.789654 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:37.789625    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nrwxd_dcf5ebb7-40c2-4efd-bd20-e79ec989bb3a/konnectivity-agent/0.log"
Sep 09 16:47:37.834280 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:37.834208    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-117.ec2.internal_12a5a478ae082f068e29cf7483267f02/haproxy/0.log"
Sep 09 16:47:41.731074 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:41.731036    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f6f59dbd4-m82tr_41cb6b67-baf5-4685-a796-f4c08b233498/authorino/0.log"
Sep 09 16:47:41.847967 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:41.847939    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-rz6hx_5afd1b8d-a975-4ea0-84da-d54d43a5e78b/kuadrant-console-plugin/0.log"
Sep 09 16:47:44.089746 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:44.089714    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7gnf2_86c73f0e-1792-412c-8fb8-294f4d6deaad/node-exporter/0.log"
Sep 09 16:47:44.110725 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:44.110683    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7gnf2_86c73f0e-1792-412c-8fb8-294f4d6deaad/kube-rbac-proxy/0.log"
Sep 09 16:47:44.131113 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:44.131086    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7gnf2_86c73f0e-1792-412c-8fb8-294f4d6deaad/init-textfile/0.log"
Sep 09 16:47:44.980405 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:44.980372    2576 scope.go:117] "RemoveContainer" containerID="3e1a377f193b6453f3742ab32090bb90ea5daa1605155a5533da549f8fdfc6ae"
Sep 09 16:47:44.988225 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:44.988202    2576 scope.go:117] "RemoveContainer" containerID="9ef00cfdde888090bd9e951b2c7cc76e4ba765ec2b838ea86369a827e6bc0dce"
Sep 09 16:47:48.215270 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.215232    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"]
Sep 09 16:47:48.215639 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.215510    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56df47ff-dbdd-4dd5-b6db-465f2a73a346" containerName="main"
Sep 09 16:47:48.215639 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.215521    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="56df47ff-dbdd-4dd5-b6db-465f2a73a346" containerName="main"
Sep 09 16:47:48.215639 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.215538    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a829b1d5-2d38-44e7-ab20-a14e402bc37e" containerName="main"
Sep 09 16:47:48.215639 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.215543    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a829b1d5-2d38-44e7-ab20-a14e402bc37e" containerName="main"
Sep 09 16:47:48.215639 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.215594    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a829b1d5-2d38-44e7-ab20-a14e402bc37e" containerName="main"
Sep 09 16:47:48.215639 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.215603    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="56df47ff-dbdd-4dd5-b6db-465f2a73a346" containerName="main"
Sep 09 16:47:48.219996 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.219974    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.222441 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.222419    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jjpd4\"/\"kube-root-ca.crt\""
Sep 09 16:47:48.223217 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.223203    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jjpd4\"/\"default-dockercfg-glm2f\""
Sep 09 16:47:48.223283 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.223264    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jjpd4\"/\"openshift-service-ca.crt\""
Sep 09 16:47:48.225472 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.225448    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"]
Sep 09 16:47:48.353164 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.353129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-proc\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.353164 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.353173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqwzm\" (UniqueName: \"kubernetes.io/projected/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-kube-api-access-cqwzm\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.353414 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.353197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-lib-modules\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.353414 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.353271    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-sys\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.353414 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.353313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-podres\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.454042 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.454006    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-proc\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.454250 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.454049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqwzm\" (UniqueName: \"kubernetes.io/projected/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-kube-api-access-cqwzm\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.454250 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.454076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-lib-modules\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.454250 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.454103    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-sys\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.454250 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.454122    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-podres\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.454250 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.454128    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-proc\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.454250 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.454245    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-podres\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.454509 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.454261    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-sys\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.454509 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.454305    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-lib-modules\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.462206 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.462185    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqwzm\" (UniqueName: \"kubernetes.io/projected/7f544bcb-c6bb-4c14-bd39-71fed2cd9827-kube-api-access-cqwzm\") pod \"perf-node-gather-daemonset-5xcl5\" (UID: \"7f544bcb-c6bb-4c14-bd39-71fed2cd9827\") " pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.531367 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.531333    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:48.804647 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:48.804623    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"]
Sep 09 16:47:48.807035 ip-10-0-137-117 kubenswrapper[2576]: W0909 16:47:48.807007    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7f544bcb_c6bb_4c14_bd39_71fed2cd9827.slice/crio-0f8b0ae97afa3e8e0cdd2a7931a2d5f77b0b84c4466a9907bab0e1e4349c3f60 WatchSource:0}: Error finding container 0f8b0ae97afa3e8e0cdd2a7931a2d5f77b0b84c4466a9907bab0e1e4349c3f60: Status 404 returned error can't find the container with id 0f8b0ae97afa3e8e0cdd2a7931a2d5f77b0b84c4466a9907bab0e1e4349c3f60
Sep 09 16:47:49.331532 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:49.331497    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5" event={"ID":"7f544bcb-c6bb-4c14-bd39-71fed2cd9827","Type":"ContainerStarted","Data":"2c270a05acf7b5d69727d9893af000170b777eabb0ee45d683723d47e442bd0e"}
Sep 09 16:47:49.331955 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:49.331549    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5" event={"ID":"7f544bcb-c6bb-4c14-bd39-71fed2cd9827","Type":"ContainerStarted","Data":"0f8b0ae97afa3e8e0cdd2a7931a2d5f77b0b84c4466a9907bab0e1e4349c3f60"}
Sep 09 16:47:49.331955 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:49.331610    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:49.349277 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:49.349228    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5" podStartSLOduration=1.349212197 podStartE2EDuration="1.349212197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:47:49.347675571 +0000 UTC m=+2345.283293998" watchObservedRunningTime="2026-09-09 16:47:49.349212197 +0000 UTC m=+2345.284830612"
Sep 09 16:47:49.787712 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:49.787673    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9s8wl_19683a66-1e9f-4c96-85bf-5291daa9d7b5/dns/0.log"
Sep 09 16:47:49.809767 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:49.809741    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9s8wl_19683a66-1e9f-4c96-85bf-5291daa9d7b5/kube-rbac-proxy/0.log"
Sep 09 16:47:49.938640 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:49.938611    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5fhrb_0e49d49c-6384-4195-8385-374e3552fc45/dns-node-resolver/0.log"
Sep 09 16:47:50.513633 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:50.513598    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5fb76c6d86-lvxfv_15c10237-6222-4e3d-bc36-6ae7eac9019e/registry/0.log"
Sep 09 16:47:50.557752 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:50.557686    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tlk4z_96e4e031-e5c2-46a3-a044-0b0d6da36fb1/node-ca/0.log"
Sep 09 16:47:51.968310 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:51.968268    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54fc5d6dc5-4qr8n_b2893fe7-94e2-4115-8362-925f3908b0a6/kube-auth-proxy/0.log"
Sep 09 16:47:52.074478 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:52.074446    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-bjd4z_24d34adc-3182-4feb-acba-41b98f7c6a8b/payload-pre-processing/0.log"
Sep 09 16:47:52.107635 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:52.107605    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-s76z5_6bcd5da5-f6c4-46aa-87dd-21844af21310/payload-processing/0.log"
Sep 09 16:47:53.029489 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:53.029456    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bmprr_f38b903a-197a-466f-87d5-e3fac326978a/serve-healthcheck-canary/0.log"
Sep 09 16:47:53.693074 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:53.693035    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lmssw_52d36234-dac2-4c5c-822c-aad4d084b857/kube-rbac-proxy/0.log"
Sep 09 16:47:53.718941 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:53.718911    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lmssw_52d36234-dac2-4c5c-822c-aad4d084b857/exporter/0.log"
Sep 09 16:47:53.740328 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:53.740304    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lmssw_52d36234-dac2-4c5c-822c-aad4d084b857/extractor/0.log"
Sep 09 16:47:55.345953 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:55.345927    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jjpd4/perf-node-gather-daemonset-5xcl5"
Sep 09 16:47:56.526840 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:56.526810    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67c7bcf47f-dflt7_17c62f0e-b8f0-4669-bce9-c37adcd58a36/manager/0.log"
Sep 09 16:47:56.543991 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:56.543964    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67c7bcf47f-dflt7_17c62f0e-b8f0-4669-bce9-c37adcd58a36/copy-manifests/0.log"
Sep 09 16:47:58.058551 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:58.058526    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7bc87557cd-pc7zb_6446a118-60ed-4da3-9f5b-1adfc9851694/manager/0.log"
Sep 09 16:47:58.850381 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:58.850347    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54fc5d6dc5-4qr8n_b2893fe7-94e2-4115-8362-925f3908b0a6/kube-auth-proxy/0.log"
Sep 09 16:47:58.930307 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:58.930278    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-bjd4z_24d34adc-3182-4feb-acba-41b98f7c6a8b/payload-pre-processing/0.log"
Sep 09 16:47:58.959053 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:47:58.959025    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-s76z5_6bcd5da5-f6c4-46aa-87dd-21844af21310/payload-processing/0.log"
Sep 09 16:48:00.893891 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:48:00.893848    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54fc5d6dc5-4qr8n_b2893fe7-94e2-4115-8362-925f3908b0a6/kube-auth-proxy/0.log"
Sep 09 16:48:00.975894 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:48:00.975865    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-bjd4z_24d34adc-3182-4feb-acba-41b98f7c6a8b/payload-pre-processing/0.log"
Sep 09 16:48:01.004231 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:48:01.004200    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-s76z5_6bcd5da5-f6c4-46aa-87dd-21844af21310/payload-processing/0.log"
Sep 09 16:48:04.650406 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:48:04.650378    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ghgq7_7466397e-16a1-4782-89b9-3f2c611e2bd0/kube-multus-additional-cni-plugins/0.log"
Sep 09 16:48:04.705975 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:48:04.705939    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ghgq7_7466397e-16a1-4782-89b9-3f2c611e2bd0/egress-router-binary-copy/0.log"
Sep 09 16:48:04.728917 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:48:04.728884    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ghgq7_7466397e-16a1-4782-89b9-3f2c611e2bd0/cni-plugins/0.log"
Sep 09 16:48:04.749992 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:48:04.749914    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ghgq7_7466397e-16a1-4782-89b9-3f2c611e2bd0/bond-cni-plugin/0.log"
Sep 09 16:48:04.770197 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:48:04.770169    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ghgq7_7466397e-16a1-4782-89b9-3f2c611e2bd0/routeoverride-cni/0.log"
Sep 09 16:48:04.792950 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:48:04.792926    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ghgq7_7466397e-16a1-4782-89b9-3f2c611e2bd0/whereabouts-cni-bincopy/0.log"
Sep 09 16:48:04.814122 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:48:04.814091    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ghgq7_7466397e-16a1-4782-89b9-3f2c611e2bd0/whereabouts-cni/0.log"
Sep 09 16:48:05.160268 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:48:05.160188    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-b9zp2_896fdb8e-35d7-4032-8b48-54f32d74e4eb/kube-multus/0.log"
Sep 09 16:48:05.342122 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:48:05.342093    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nwfs2_03652816-9d0a-45d3-9390-04cc6acc8cee/network-metrics-daemon/0.log"
Sep 09 16:48:05.360884 ip-10-0-137-117 kubenswrapper[2576]: I0909 16:48:05.360861    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nwfs2_03652816-9d0a-45d3-9390-04cc6acc8cee/kube-rbac-proxy/0.log"