Sep 10 16:15:31.265646 ip-10-0-138-74 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 16:15:31.265659 ip-10-0-138-74 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 16:15:31.265669 ip-10-0-138-74 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 16:15:31.265872 ip-10-0-138-74 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 16:15:42.440256 ip-10-0-138-74 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 16:15:42.440271 ip-10-0-138-74 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3138d71d9dfd40aeae6d070e389f031d --
Sep 10 16:17:21.689816 ip-10-0-138-74 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 16:17:22.103522 ip-10-0-138-74 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 16:17:22.103522 ip-10-0-138-74 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 16:17:22.103522 ip-10-0-138-74 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 16:17:22.103522 ip-10-0-138-74 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 16:17:22.103522 ip-10-0-138-74 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 16:17:22.105358 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.105275    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 16:17:22.109072 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109057    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 16:17:22.109072 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109071    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109075    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109079    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109083    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109085    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109088    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109091    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109094    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109096    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109099    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109102    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109105    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109107    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109110    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109113    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109115    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109118    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109121    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109123    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109127    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 16:17:22.109137 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109129    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109132    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109134    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109137    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109140    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109143    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109145    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109148    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109151    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109154    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109157    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109159    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109162    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109165    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109167    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109169    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109172    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109174    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109177    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 16:17:22.109635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109179    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109183    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109186    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109188    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109191    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109193    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109196    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109198    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109201    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109203    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109206    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109208    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109211    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109213    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109216    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109219    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109221    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109224    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109226    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109229    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 16:17:22.110105 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109232    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109235    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109237    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109241    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109245    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109248    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109251    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109254    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109256    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109259    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109263    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109267    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109270    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109275    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109279    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109283    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109286    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109289    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109292    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 16:17:22.110600 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109295    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109297    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109300    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109303    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109305    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109308    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109311    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109314    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109316    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109319    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109322    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109324    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109327    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109329    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109332    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109335    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109337    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109339    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109342    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.109344    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 16:17:22.111107 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110066    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110074    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110078    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110082    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110085    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110087    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110090    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110093    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110096    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110099    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110101    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110104    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110106    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110109    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110112    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110115    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110118    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110121    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110123    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110126    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 16:17:22.111622 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110129    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110132    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110135    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110137    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110140    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110143    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110145    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110148    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110150    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110153    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110155    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110158    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110161    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110163    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110166    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110168    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110171    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110173    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110176    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110178    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 16:17:22.112124 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110181    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110183    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110186    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110189    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110192    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110194    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110197    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110199    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110201    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110204    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110207    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110210    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110215    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110217    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110220    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110222    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110225    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110227    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110231    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110233    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 16:17:22.112640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110236    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110238    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110240    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110243    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110245    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110248    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110250    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110253    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110255    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110258    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110260    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110263    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110265    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110268    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110271    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110274    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110279    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110282    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110285    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 16:17:22.113140 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110288    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110291    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110293    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110296    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110300    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110304    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110307    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110309    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110312    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110315    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110317    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110320    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110322    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110325    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110327    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110330    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110332    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110335    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110337    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 16:17:22.113628 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110340    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110693    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110698    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110701    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110704    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110707    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110711    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110715    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110718    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110722    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110725    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110727    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110731    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110734    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110736    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110739    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110742    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110745    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110748    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110750    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 16:17:22.114086 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110753    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110755    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110758    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110760    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110763    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110766    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110768    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110771    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110774    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110776    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110779    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110781    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110784    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110786    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110789    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110792    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110795    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110797    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110800    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 16:17:22.114575 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110802    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110805    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110808    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110810    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110813    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110815    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110818    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110820    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110823    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110825    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110828    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110831    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110834    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110836    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110839    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110841    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110844    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110848    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110852    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110855    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 16:17:22.115049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110857    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110861    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110864    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110866    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110869    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110872    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110877    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110879    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110882    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110884    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110887    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110891    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110894    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110897    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110900    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110903    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110906    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110908    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110911    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 16:17:22.115595 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110913    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110916    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110918    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110922    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110924    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110927    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110929    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110932    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110934    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110937    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110939    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110942    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110944    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110947    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110949    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110952    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110955    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110957    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110959    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110966    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 16:17:22.116100 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110968    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.110971    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111040    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111075    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111081    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111086    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111089    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111093    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111097    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111100    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111104    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111107    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111110    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111114    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111117    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111121    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111124    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111127    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111130    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111132    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111136    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111139    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111142    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111145    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 16:17:22.116610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111148    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111152    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111155    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111158    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111161    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111164    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111167    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111172    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111175    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111178    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111181    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111185    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111187    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111190    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111193    2580 flags.go:64] FLAG: --enable-server="true"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111196    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111200    2580 flags.go:64] FLAG: --event-burst="100"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111204    2580 flags.go:64] FLAG: --event-qps="50"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111207    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111211    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111214    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111217    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111220    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111224    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111227    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 10 16:17:22.117183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111229    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111233    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111235    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111238    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111241    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111244    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111247    2580 flags.go:64] FLAG: --feature-gates=""
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111250    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111253    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111256    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111259    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111262    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111265    2580 flags.go:64] FLAG: --help="false"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111268    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111273    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111276    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111279    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111282    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111285    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111288    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111291    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111294    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111296    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 16:17:22.118004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111299    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111302    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111305    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111308    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111311    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111314    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111317    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111320    2580 flags.go:64] FLAG: --lock-file=""
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111323    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111326    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111329    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111334    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111337    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111340    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111342    2580 flags.go:64] FLAG: --logging-format="text"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111345    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111350    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111352    2580 flags.go:64] FLAG: --manifest-url=""
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111355    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111359    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111362    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111366    2580 flags.go:64] FLAG: --max-pods="110"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111369    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111373    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111376    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 16:17:22.119020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111379    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111382    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111385    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111387    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111394    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111397    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111400    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111403    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111406    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111411    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111414    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111417    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111422    2580 flags.go:64] FLAG: --port="10250"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111425    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111428    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03624bb85f95819fc"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111431    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111434    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111437    2580 flags.go:64] FLAG: --register-node="true"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111440    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111443    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111446    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111449    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111452    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111459    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111462    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 16:17:22.120105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111465    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111468    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111471    2580 flags.go:64] FLAG: --runonce="false"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111474    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111477    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111480    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111485    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111487    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111490    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111493    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111496    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111499    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111502    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111505    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111508    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111510    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111513    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111516    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111522    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111525    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111540    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111545    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111548    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111551    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111554    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 16:17:22.120936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111556    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111559    2580 flags.go:64] FLAG: --v="2"
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111568    2580 flags.go:64] FLAG: --version="false"
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111579    2580 flags.go:64] FLAG: --vmodule=""
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111582    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.111587    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111693    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111697    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111700    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111703    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111706    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111708    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111711    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111715    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111718    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111721    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111723    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111726    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111730    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111733    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 16:17:22.121640 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111737    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111740    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111743    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111746    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111749    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111751    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111754    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111756    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111759    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111761    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111764    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111767    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111769    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111772    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111774    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111777    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111779    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111783    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111786    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111788    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 16:17:22.122185 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111791    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111793    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111795    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111798    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111801    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111804    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111808    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111812    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111815    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111818    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111821    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111824    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111826    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111829    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111832    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111835    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111837    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111840    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111842    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111845    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 16:17:22.122696 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111847    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111850    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111853    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111855    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111858    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111860    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111863    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111865    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111868    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111872    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111874    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111877    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111879    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111882    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111884    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111886    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111889    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111892    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111895    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111898    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 16:17:22.123194 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111900    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111903    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111905    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111908    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111910    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111914    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111917    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111920    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111923    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111925    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111928    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111930    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111933    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111935    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111938    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111941    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111943    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111946    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111948    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111951    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 16:17:22.123785 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111953    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 16:17:22.124322 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111956    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 16:17:22.124322 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111959    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 16:17:22.124322 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111962    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 16:17:22.124322 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.111964    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 16:17:22.124322 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.112513    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 16:17:22.124322 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.120330    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 16:17:22.124322 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.120385    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 16:17:22.124322 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.121485    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 16:17:22.124322 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.121524    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 16:17:22.124322 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121817    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 16:17:22.124322 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121832    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 16:17:22.124322 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121836    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 16:17:22.124322 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121839    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 16:17:22.124322 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121842    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121845    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121848    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121852    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121855    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121857    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121860    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121863    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121866    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121869    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121872    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121874    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121877    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121880    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121882    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121885    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121888    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121890    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121893    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 16:17:22.124698 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121895    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121898    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121901    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121904    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121906    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121909    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121911    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121915    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121918    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121920    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121924    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121927    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121930    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121933    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121935    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121938    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121940    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121943    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121945    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 16:17:22.125153 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121948    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121951    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121953    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121956    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121958    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121961    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121963    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121966    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121968    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121971    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121973    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121976    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121979    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121981    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121984    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121986    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121988    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121991    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121994    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.121997    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 16:17:22.125618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122001    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122003    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122006    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122009    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122012    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122015    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122017    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122020    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122022    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122025    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122027    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122030    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122032    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122035    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122037    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122041    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122045    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122048    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122051    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122054    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 16:17:22.126101 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122056    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122060    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122062    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122066    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122069    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122072    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122075    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122078    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122082    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122085    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122088    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122090    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122093    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122095    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122098    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122102    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 16:17:22.126605 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122106    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 16:17:22.127011 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.122112    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 16:17:22.127011 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122223    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 16:17:22.127011 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122227    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 16:17:22.127011 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122230    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 16:17:22.127011 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122232    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 16:17:22.127011 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122235    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 16:17:22.127011 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122238    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 16:17:22.127011 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122240    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 16:17:22.127011 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122243    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 16:17:22.127011 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122245    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 16:17:22.127011 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122248    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 16:17:22.127011 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122250    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 16:17:22.127011 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122253    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 16:17:22.127011 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122257    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122261    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122263    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122266    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122269    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122271    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122274    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122277    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122279    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122282    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122284    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122287    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122289    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122291    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122294    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122296    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122299    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122302    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122304    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122307    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 16:17:22.127352 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122309    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122312    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122316    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122318    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122321    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122324    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122326    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122328    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122331    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122334    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122336    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122339    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122341    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122345    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122347    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122350    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122352    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122355    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122357    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 16:17:22.127934 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122361    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122364    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122367    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122369    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122372    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122375    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122377    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122380    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122382    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122385    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122388    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122390    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122393    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122396    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122398    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122401    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122403    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122406    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122408    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122411    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 16:17:22.128410 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122413    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122416    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122418    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122421    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122423    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122426    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122429    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122432    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122435    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122437    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122440    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122442    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122445    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122447    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122449    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122452    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122454    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122457    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122459    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122462    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 16:17:22.128915 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122464    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 16:17:22.129456 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122467    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 16:17:22.129456 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122470    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 16:17:22.129456 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122472    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 16:17:22.129456 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122475    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 16:17:22.129456 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122477    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 16:17:22.129456 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122480    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 16:17:22.129456 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:22.122482    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 16:17:22.129456 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.122487    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 16:17:22.129456 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.122606    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 16:17:22.129456 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.124618    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 16:17:22.129456 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.125504    2580 server.go:1019] "Starting client certificate rotation"
Sep 10 16:17:22.129456 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.125560    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 16:17:22.129456 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.125595    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 16:17:22.129837 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.128178    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 16:17:22.129837 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.129267    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 16:17:22.137131 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.137116    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 10 16:17:22.139961 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.139947    2580 log.go:25] "Validated CRI v1 image API"
Sep 10 16:17:22.140816 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.140799    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 16:17:22.147183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.147166    2580 fs.go:135] Filesystem UUIDs: map[2be05b63-3a28-4a6f-a608-c74d7fdcb742:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e942e671-032f-411d-b935-f231f0debe5c:/dev/nvme0n1p3]
Sep 10 16:17:22.147252 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.147183    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 16:17:22.151947 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.151834    2580 manager.go:216] Machine: {Timestamp:2026-09-10 16:17:22.150685995 +0000 UTC m=+0.351728399 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097754 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21be3bd95f7e7add9732731644fe6a SystemUUID:ec21be3b-d95f-7e7a-dd97-32731644fe6a BootID:3138d71d-9dfd-40ae-ae6d-070e389f031d Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d8:d8:d6:38:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d8:d8:d6:38:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:35:58:ad:9e:9a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 16:17:22.151947 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.151940    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 16:17:22.152051 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.152012    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 16:17:22.154088 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.154064    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 16:17:22.154227 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.154090    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-74.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 16:17:22.155451 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.155436    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 16:17:22.155451 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.155451    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 16:17:22.155581 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.155460    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 16:17:22.157049 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.157031    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 16:17:22.157114 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.157104    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 16:17:22.158100 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.158088    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 16:17:22.162642 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.162631    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 16:17:22.162765 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.162755    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 16:17:22.162806 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.162783    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 16:17:22.162806 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.162799    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 16:17:22.162876 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.162810    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 16:17:22.162876 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.162819    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 10 16:17:22.162876 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.162826    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 16:17:22.163349 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.163328    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 16:17:22.163446 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.163354    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 16:17:22.167693 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.167671    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 16:17:22.170062 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.170049    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 16:17:22.170125 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.170078    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 16:17:22.171513 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.171502    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 16:17:22.171575 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.171517    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 16:17:22.171575 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.171523    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 16:17:22.171575 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.171542    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 16:17:22.171575 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.171551    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 16:17:22.171575 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.171556    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 16:17:22.171796 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.171786    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 16:17:22.171825 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.171798    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 16:17:22.171825 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.171808    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 16:17:22.171825 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.171817    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 16:17:22.171896 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.171829    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 16:17:22.171896 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.171841    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 16:17:22.174391 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.174367    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 16:17:22.174435 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.174372    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 16:17:22.175611 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.175598    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 16:17:22.175645 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.175613    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 16:17:22.179198 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.179185    2580 server.go:1268] "Started kubelet"
Sep 10 16:17:22.179302 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.179262    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 16:17:22.179424 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.179264    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 16:17:22.179472 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.179435    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 16:17:22.180184 ip-10-0-138-74 systemd[1]: Started Kubernetes Kubelet.
Sep 10 16:17:22.180577 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.180558    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 16:17:22.183583 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.183550    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 16:17:22.184436 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.184248    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 16:17:22.185183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.185167    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 16:17:22.185262 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.185183    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 16:17:22.185328 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.185264    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 10 16:17:22.185884 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.185777    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 16:17:22.185884 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.185795    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 16:17:22.186019 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.185890    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 16:17:22.186019 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.185981    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 16:17:22.186019 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.185988    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 16:17:22.186337 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.186318    2580 factory.go:55] Registering systemd factory
Sep 10 16:17:22.186418 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.186351    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 10 16:17:22.186629 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.186614    2580 factory.go:153] Registering CRI-O factory
Sep 10 16:17:22.186629 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.186629    2580 factory.go:223] Registration of the crio container factory successfully
Sep 10 16:17:22.186747 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.186689    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 16:17:22.186747 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.186706    2580 factory.go:103] Registering Raw factory
Sep 10 16:17:22.186747 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.186721    2580 manager.go:1196] Started watching for new ooms in manager
Sep 10 16:17:22.187171 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.187157    2580 manager.go:349] Starting recovery of all containers
Sep 10 16:17:22.188803 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.188774    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 16:17:22.188936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.188919    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-74.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 16:17:22.189054 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.189010    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 16:17:22.189897 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.189005    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-74.ec2.internal.18d401e6a9ffea0c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-74.ec2.internal,UID:ip-10-0-138-74.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-74.ec2.internal,},FirstTimestamp:2026-09-10 16:17:22.179156492 +0000 UTC m=+0.380198893,LastTimestamp:2026-09-10 16:17:22.179156492 +0000 UTC m=+0.380198893,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-74.ec2.internal,}"
Sep 10 16:17:22.192408 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.192389    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gk5hf"
Sep 10 16:17:22.197624 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.197601    2580 manager.go:354] Recovery completed
Sep 10 16:17:22.198755 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.198731    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gk5hf"
Sep 10 16:17:22.203651 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.203634    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 16:17:22.205292 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.205278    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 16:17:22.205355 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.205304    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 16:17:22.205355 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.205313    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID"
Sep 10 16:17:22.205810 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.205795    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 16:17:22.205870 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.205810    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 16:17:22.205870 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.205812    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 16:17:22.205870 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.205828    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 16:17:22.206448 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.206434    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 16:17:22.206498 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.206462    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 16:17:22.206498 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.206476    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID"
Sep 10 16:17:22.207132 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.207036    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-74.ec2.internal.18d401e6ab8eb5a3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-74.ec2.internal,UID:ip-10-0-138-74.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-74.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-74.ec2.internal,},FirstTimestamp:2026-09-10 16:17:22.205291939 +0000 UTC m=+0.406334340,LastTimestamp:2026-09-10 16:17:22.205291939 +0000 UTC m=+0.406334340,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-74.ec2.internal,}"
Sep 10 16:17:22.207956 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.207862    2580 policy_none.go:49] "None policy: Start"
Sep 10 16:17:22.208018 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.207960    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 16:17:22.208018 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.207970    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 16:17:22.209912 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.209901    2580 policy_none.go:47] "Start"
Sep 10 16:17:22.234207 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.234190    2580 manager.go:337] "Starting Device Plugin manager"
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.234230    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.234243    2580 server.go:87] "Starting device plugin registration server"
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.234519    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.234559    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.234641    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.234735    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.234744    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.235186    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.235232    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.246907    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.248075    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.248114    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.248131    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.248196    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 16:17:22.257541 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.250340    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 16:17:22.335593 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.335572    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 16:17:22.336697 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.336680    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 16:17:22.336763 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.336708    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 16:17:22.336763 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.336719    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID"
Sep 10 16:17:22.336763 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.336737    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.342879 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.342865    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.342940 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.342884    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-74.ec2.internal\": node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 10 16:17:22.348261 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.348246    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"]
Sep 10 16:17:22.348335 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.348298    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 16:17:22.349074 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.349051    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 16:17:22.349175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.349080    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 16:17:22.349175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.349094    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID"
Sep 10 16:17:22.351347 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.351333    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 16:17:22.351463 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.351443    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.351510 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.351480    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 16:17:22.352119 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.352102    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 16:17:22.352186 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.352129    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 16:17:22.352186 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.352143    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID"
Sep 10 16:17:22.352186 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.352105    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 16:17:22.352325 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.352213    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 16:17:22.352325 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.352231    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID"
Sep 10 16:17:22.354397 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.354345    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.354397 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.354377    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 16:17:22.355306 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.355289    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 16:17:22.355389 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.355318    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 16:17:22.355389 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.355331    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID"
Sep 10 16:17:22.358403 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.358387    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 10 16:17:22.378056 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.378040    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-74.ec2.internal\" not found" node="ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.382406 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.382391    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-74.ec2.internal\" not found" node="ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.458832 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.458802    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 10 16:17:22.486600 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.486575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18ee5e215f5e8ba4676df80e0d119df6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-74.ec2.internal\" (UID: \"18ee5e215f5e8ba4676df80e0d119df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.486690 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.486619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/015c0be4345cf7cdba6b30c6e8b201df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"015c0be4345cf7cdba6b30c6e8b201df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.486690 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.486664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/015c0be4345cf7cdba6b30c6e8b201df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"015c0be4345cf7cdba6b30c6e8b201df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.559786 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.559757    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 10 16:17:22.587034 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.587011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18ee5e215f5e8ba4676df80e0d119df6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-74.ec2.internal\" (UID: \"18ee5e215f5e8ba4676df80e0d119df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.587140 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.587038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/015c0be4345cf7cdba6b30c6e8b201df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"015c0be4345cf7cdba6b30c6e8b201df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.587140 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.587056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/015c0be4345cf7cdba6b30c6e8b201df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"015c0be4345cf7cdba6b30c6e8b201df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.587140 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.587086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/015c0be4345cf7cdba6b30c6e8b201df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"015c0be4345cf7cdba6b30c6e8b201df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.587140 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.587101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/015c0be4345cf7cdba6b30c6e8b201df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"015c0be4345cf7cdba6b30c6e8b201df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.587140 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.587100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18ee5e215f5e8ba4676df80e0d119df6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-74.ec2.internal\" (UID: \"18ee5e215f5e8ba4676df80e0d119df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.660366 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.660282    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 10 16:17:22.680608 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.680573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.687766 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:22.687745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:22.761053 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.761015    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 10 16:17:22.861552 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.861501    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 10 16:17:22.962046 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:22.961976    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 10 16:17:23.065208 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:23.065169    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 10 16:17:23.127895 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:23.127859    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 16:17:23.128469 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:23.128019    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 16:17:23.166322 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:23.166295    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 10 16:17:23.188401 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:23.188370    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 16:17:23.203530 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:23.203486    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 16:12:22 +0000 UTC" deadline="2028-02-25 21:03:46.909272109 +0000 UTC"
Sep 10 16:17:23.203530 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:23.203526    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12796h46m23.7057499s"
Sep 10 16:17:23.286488 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:23.286452    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:23.297062 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:23.297036    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 16:17:23.298867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:23.298848    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 10 16:17:23.308401 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:23.308381    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 16:17:23.382166 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:23.382143    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 16:17:23.413556 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:23.413513    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015c0be4345cf7cdba6b30c6e8b201df.slice/crio-5dd545e73844eb8fb256af38ef30fb00503798d538f3caacfb73d793647d477e WatchSource:0}: Error finding container 5dd545e73844eb8fb256af38ef30fb00503798d538f3caacfb73d793647d477e: Status 404 returned error can't find the container with id 5dd545e73844eb8fb256af38ef30fb00503798d538f3caacfb73d793647d477e
Sep 10 16:17:23.414540 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:23.414512    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ee5e215f5e8ba4676df80e0d119df6.slice/crio-9cd334304405a9a8c6480a2bac559ed497f118504ad5a75a6f467d46769b7c5e WatchSource:0}: Error finding container 9cd334304405a9a8c6480a2bac559ed497f118504ad5a75a6f467d46769b7c5e: Status 404 returned error can't find the container with id 9cd334304405a9a8c6480a2bac559ed497f118504ad5a75a6f467d46769b7c5e
Sep 10 16:17:23.418558 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:23.418543    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 16:17:23.635114 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:23.635042    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 16:17:23.948132 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:23.947899    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 16:17:24.164454 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.164420    2580 apiserver.go:52] "Watching apiserver"
Sep 10 16:17:24.169987 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.169961    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 16:17:24.170427 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.170401    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-95jqx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal","openshift-multus/multus-72nfz","openshift-network-operator/iptables-alerter-wdd9f","kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w","openshift-dns/node-resolver-pgclc","openshift-image-registry/node-ca-2zpjw","openshift-multus/multus-additional-cni-plugins-t4wn7","openshift-multus/network-metrics-daemon-lnx4z","openshift-network-diagnostics/network-check-target-vzlvs","openshift-ovn-kubernetes/ovnkube-node-mbjsw","kube-system/global-pull-secret-syncer-dc7jq","kube-system/konnectivity-agent-5vf27"]
Sep 10 16:17:24.173162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.173136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.175865 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.175834    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.176314 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.176059    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 16:17:24.176314 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.176074    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 16:17:24.176314 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.176066    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gqgs9\""
Sep 10 16:17:24.178181 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.177993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wdd9f"
Sep 10 16:17:24.179400 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.178891    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 16:17:24.179400 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.178949    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-p25v8\""
Sep 10 16:17:24.179400 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.179119    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 16:17:24.179400 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.179187    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 16:17:24.179400 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.179242    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 16:17:24.179971 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.179950    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 16:17:24.180379 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.180348    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.180915 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.180479    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hmg75\""
Sep 10 16:17:24.180915 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.180775    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 16:17:24.181066 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.181030    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 16:17:24.182619 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.182599    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 16:17:24.184047 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.182757    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8llsm\""
Sep 10 16:17:24.184047 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.182813    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 16:17:24.184047 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.182941    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 16:17:24.184047 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.183496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pgclc"
Sep 10 16:17:24.185197 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.185178    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2zpjw"
Sep 10 16:17:24.185782 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.185763    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 16:17:24.186926 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.186898    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 16:17:24.187096 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.187077    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 16:17:24.187165 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.187148    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pbklj\""
Sep 10 16:17:24.187629 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.187518    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 16:17:24.187718 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.187691    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 16:17:24.188209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.187836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.188912 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.188685    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 16:17:24.188912 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.188754    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r7t2d\""
Sep 10 16:17:24.190124 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.190064    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 16:17:24.191068 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.191045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:24.191170 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.191140    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:24.192302 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.192281    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sqrnb\""
Sep 10 16:17:24.192738 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.192716    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 16:17:24.195818 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.195733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-device-dir\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.195818 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.195769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-etc-selinux\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.195982 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.195824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srqz4\" (UniqueName: \"kubernetes.io/projected/5f1f8695-2093-47a4-8397-730fcd19cedc-kube-api-access-srqz4\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.195982 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.195860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8-tmp-dir\") pod \"node-resolver-pgclc\" (UID: \"2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8\") " pod="openshift-dns/node-resolver-pgclc"
Sep 10 16:17:24.195982 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.195909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-run-netns\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.195982 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.195949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-run-multus-certs\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.195982 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.195977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-689xq\" (UniqueName: \"kubernetes.io/projected/87b01bad-5f40-4c30-aa70-bfd96d239873-kube-api-access-689xq\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-sysconfig\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196044    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-tuned\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-system-cni-dir\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-multus-cni-dir\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-cnibin\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-multus-conf-dir\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-multus-daemon-config\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-etc-kubernetes\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmjq\" (UniqueName: \"kubernetes.io/projected/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-kube-api-access-jvmjq\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-var-lib-cni-multus\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-var-lib-kubelet\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.196569    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87b01bad-5f40-4c30-aa70-bfd96d239873-system-cni-dir\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87b01bad-5f40-4c30-aa70-bfd96d239873-cnibin\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.197209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/87b01bad-5f40-4c30-aa70-bfd96d239873-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/87b01bad-5f40-4c30-aa70-bfd96d239873-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d95286cd-ebcd-47c2-9a95-d51f5e62576e-host\") pod \"node-ca-2zpjw\" (UID: \"d95286cd-ebcd-47c2-9a95-d51f5e62576e\") " pod="openshift-image-registry/node-ca-2zpjw"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qctr6\" (UniqueName: \"kubernetes.io/projected/d95286cd-ebcd-47c2-9a95-d51f5e62576e-kube-api-access-qctr6\") pod \"node-ca-2zpjw\" (UID: \"d95286cd-ebcd-47c2-9a95-d51f5e62576e\") " pod="openshift-image-registry/node-ca-2zpjw"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87b01bad-5f40-4c30-aa70-bfd96d239873-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-kubernetes\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-run\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-lib-modules\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196874    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc32931b-ee56-4382-96bb-2bbb378d6115-tmp\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-socket-dir\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d95286cd-ebcd-47c2-9a95-d51f5e62576e-serviceca\") pod \"node-ca-2zpjw\" (UID: \"d95286cd-ebcd-47c2-9a95-d51f5e62576e\") " pod="openshift-image-registry/node-ca-2zpjw"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.196955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-var-lib-cni-bin\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87b01bad-5f40-4c30-aa70-bfd96d239873-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-modprobe-d\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197052    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-sysctl-d\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-sysctl-conf\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.198164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ed41c9e-f856-44a3-b13d-b6944bbfa63b-host-slash\") pod \"iptables-alerter-wdd9f\" (UID: \"0ed41c9e-f856-44a3-b13d-b6944bbfa63b\") " pod="openshift-network-operator/iptables-alerter-wdd9f"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tj95\" (UniqueName: \"kubernetes.io/projected/0ed41c9e-f856-44a3-b13d-b6944bbfa63b-kube-api-access-4tj95\") pod \"iptables-alerter-wdd9f\" (UID: \"0ed41c9e-f856-44a3-b13d-b6944bbfa63b\") " pod="openshift-network-operator/iptables-alerter-wdd9f"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-cni-binary-copy\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-systemd\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-sys\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppcdb\" (UniqueName: \"kubernetes.io/projected/cc32931b-ee56-4382-96bb-2bbb378d6115-kube-api-access-ppcdb\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0ed41c9e-f856-44a3-b13d-b6944bbfa63b-iptables-alerter-script\") pod \"iptables-alerter-wdd9f\" (UID: \"0ed41c9e-f856-44a3-b13d-b6944bbfa63b\") " pod="openshift-network-operator/iptables-alerter-wdd9f"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-run-k8s-cni-cncf-io\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-host\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-registration-dir\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-sys-fs\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8-hosts-file\") pod \"node-resolver-pgclc\" (UID: \"2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8\") " pod="openshift-dns/node-resolver-pgclc"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb55l\" (UniqueName: \"kubernetes.io/projected/2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8-kube-api-access-zb55l\") pod \"node-resolver-pgclc\" (UID: \"2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8\") " pod="openshift-dns/node-resolver-pgclc"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-multus-socket-dir-parent\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197644    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-hostroot\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.199020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.197678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-var-lib-kubelet\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.199757 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.199449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.202127 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.202106    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:24.202236 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.202173    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:24.202770 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.202301    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 16:17:24.202770 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.202362    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 16:17:24.202770 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.202536    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 16:17:24.202770 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.202551    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 16:17:24.202770 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.202632    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 16:17:24.202770 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.202736    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 16:17:24.202770 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.202753    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c5t4r\""
Sep 10 16:17:24.203167 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.202918    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 16:17:24.204754 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.204732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5vf27"
Sep 10 16:17:24.206765 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.206742    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 16:17:24.207212 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.207068    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4vz7z\""
Sep 10 16:17:24.207212 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.207193    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 16:17:24.231311 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.231283    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ff4b6"
Sep 10 16:17:24.239733 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.239714    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ff4b6"
Sep 10 16:17:24.256062 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.255976    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" event={"ID":"18ee5e215f5e8ba4676df80e0d119df6","Type":"ContainerStarted","Data":"9cd334304405a9a8c6480a2bac559ed497f118504ad5a75a6f467d46769b7c5e"}
Sep 10 16:17:24.258373 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.258345    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" event={"ID":"015c0be4345cf7cdba6b30c6e8b201df","Type":"ContainerStarted","Data":"5dd545e73844eb8fb256af38ef30fb00503798d538f3caacfb73d793647d477e"}
Sep 10 16:17:24.286977 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.286956    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 16:17:24.298391 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-multus-conf-dir\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.298483 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-multus-daemon-config\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.298483 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/87b01bad-5f40-4c30-aa70-bfd96d239873-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.298483 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/87b01bad-5f40-4c30-aa70-bfd96d239873-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.298483 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ed41c9e-f856-44a3-b13d-b6944bbfa63b-host-slash\") pod \"iptables-alerter-wdd9f\" (UID: \"0ed41c9e-f856-44a3-b13d-b6944bbfa63b\") " pod="openshift-network-operator/iptables-alerter-wdd9f"
Sep 10 16:17:24.298483 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-multus-conf-dir\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.298817 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tj95\" (UniqueName: \"kubernetes.io/projected/0ed41c9e-f856-44a3-b13d-b6944bbfa63b-kube-api-access-4tj95\") pod \"iptables-alerter-wdd9f\" (UID: \"0ed41c9e-f856-44a3-b13d-b6944bbfa63b\") " pod="openshift-network-operator/iptables-alerter-wdd9f"
Sep 10 16:17:24.298817 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-log-socket\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.298817 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-var-lib-cni-multus\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.298817 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-var-lib-kubelet\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.298817 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87b01bad-5f40-4c30-aa70-bfd96d239873-cnibin\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.298817 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87b01bad-5f40-4c30-aa70-bfd96d239873-cnibin\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.298817 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-var-lib-cni-multus\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.298817 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298765    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-var-lib-kubelet\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.298817 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ed41c9e-f856-44a3-b13d-b6944bbfa63b-host-slash\") pod \"iptables-alerter-wdd9f\" (UID: \"0ed41c9e-f856-44a3-b13d-b6944bbfa63b\") " pod="openshift-network-operator/iptables-alerter-wdd9f"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-kubernetes\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-socket-dir\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-cni-bin\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qctr6\" (UniqueName: \"kubernetes.io/projected/d95286cd-ebcd-47c2-9a95-d51f5e62576e-kube-api-access-qctr6\") pod \"node-ca-2zpjw\" (UID: \"d95286cd-ebcd-47c2-9a95-d51f5e62576e\") " pod="openshift-image-registry/node-ca-2zpjw"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.298999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87b01bad-5f40-4c30-aa70-bfd96d239873-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-kubernetes\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-run\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-socket-dir\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-var-lib-openvswitch\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-var-lib-cni-bin\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299149    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-run\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299150    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/87b01bad-5f40-4c30-aa70-bfd96d239873-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-var-lib-cni-bin\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-multus-daemon-config\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-sysctl-conf\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299265    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25606248-47cf-4de0-aab7-48d1123e9901-dbus\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:24.299295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299295    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d3fba85-1986-4255-befb-90e94d9aa2e4-env-overrides\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-sysctl-conf\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6smd\" (UniqueName: \"kubernetes.io/projected/4d3fba85-1986-4255-befb-90e94d9aa2e4-kube-api-access-d6smd\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-cni-binary-copy\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-sys\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppcdb\" (UniqueName: \"kubernetes.io/projected/cc32931b-ee56-4382-96bb-2bbb378d6115-kube-api-access-ppcdb\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0ed41c9e-f856-44a3-b13d-b6944bbfa63b-iptables-alerter-script\") pod \"iptables-alerter-wdd9f\" (UID: \"0ed41c9e-f856-44a3-b13d-b6944bbfa63b\") " pod="openshift-network-operator/iptables-alerter-wdd9f"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvpwq\" (UniqueName: \"kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq\") pod \"network-check-target-vzlvs\" (UID: \"218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f\") " pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-sys\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25606248-47cf-4de0-aab7-48d1123e9901-kubelet-config\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-run-systemd\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/87b01bad-5f40-4c30-aa70-bfd96d239873-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-etc-openvswitch\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-host\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb55l\" (UniqueName: \"kubernetes.io/projected/2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8-kube-api-access-zb55l\") pod \"node-resolver-pgclc\" (UID: \"2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8\") " pod="openshift-dns/node-resolver-pgclc"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-kubelet\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87b01bad-5f40-4c30-aa70-bfd96d239873-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.300175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-hostroot\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-var-lib-kubelet\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-host\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-device-dir\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-etc-selinux\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-hostroot\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-cni-netd\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-sysconfig\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-var-lib-kubelet\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-tuned\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-device-dir\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299904    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-cni-binary-copy\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299927    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-etc-selinux\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-multus-cni-dir\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-sysconfig\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-cnibin\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-etc-kubernetes\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.299998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0ed41c9e-f856-44a3-b13d-b6944bbfa63b-iptables-alerter-script\") pod \"iptables-alerter-wdd9f\" (UID: \"0ed41c9e-f856-44a3-b13d-b6944bbfa63b\") " pod="openshift-network-operator/iptables-alerter-wdd9f"
Sep 10 16:17:24.301025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-multus-cni-dir\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmjq\" (UniqueName: \"kubernetes.io/projected/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-kube-api-access-jvmjq\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-etc-kubernetes\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300061    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-cnibin\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87b01bad-5f40-4c30-aa70-bfd96d239873-system-cni-dir\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srqz4\" (UniqueName: \"kubernetes.io/projected/5f1f8695-2093-47a4-8397-730fcd19cedc-kube-api-access-srqz4\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87b01bad-5f40-4c30-aa70-bfd96d239873-system-cni-dir\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8-tmp-dir\") pod \"node-resolver-pgclc\" (UID: \"2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8\") " pod="openshift-dns/node-resolver-pgclc"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d3fba85-1986-4255-befb-90e94d9aa2e4-ovn-node-metrics-cert\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d95286cd-ebcd-47c2-9a95-d51f5e62576e-host\") pod \"node-ca-2zpjw\" (UID: \"d95286cd-ebcd-47c2-9a95-d51f5e62576e\") " pod="openshift-image-registry/node-ca-2zpjw"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-lib-modules\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc32931b-ee56-4382-96bb-2bbb378d6115-tmp\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300307    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d95286cd-ebcd-47c2-9a95-d51f5e62576e-host\") pod \"node-ca-2zpjw\" (UID: \"d95286cd-ebcd-47c2-9a95-d51f5e62576e\") " pod="openshift-image-registry/node-ca-2zpjw"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300362    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-sys-fs\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-sys-fs\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8-hosts-file\") pod \"node-resolver-pgclc\" (UID: \"2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8\") " pod="openshift-dns/node-resolver-pgclc"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300452    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-lib-modules\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.301867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8-hosts-file\") pod \"node-resolver-pgclc\" (UID: \"2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8\") " pod="openshift-dns/node-resolver-pgclc"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d95286cd-ebcd-47c2-9a95-d51f5e62576e-serviceca\") pod \"node-ca-2zpjw\" (UID: \"d95286cd-ebcd-47c2-9a95-d51f5e62576e\") " pod="openshift-image-registry/node-ca-2zpjw"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8-tmp-dir\") pod \"node-resolver-pgclc\" (UID: \"2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8\") " pod="openshift-dns/node-resolver-pgclc"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87b01bad-5f40-4c30-aa70-bfd96d239873-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-modprobe-d\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-sysctl-d\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-systemd-units\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d3fba85-1986-4255-befb-90e94d9aa2e4-ovnkube-script-lib\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-systemd\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-modprobe-d\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87b01bad-5f40-4c30-aa70-bfd96d239873-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-sysctl-d\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-systemd\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-run-netns\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.302703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-run-openvswitch\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-run-k8s-cni-cncf-io\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-registration-dir\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs\") pod \"network-metrics-daemon-lnx4z\" (UID: \"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10\") " pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvb2q\" (UniqueName: \"kubernetes.io/projected/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-kube-api-access-gvb2q\") pod \"network-metrics-daemon-lnx4z\" (UID: \"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10\") " pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-run-k8s-cni-cncf-io\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-slash\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d95286cd-ebcd-47c2-9a95-d51f5e62576e-serviceca\") pod \"node-ca-2zpjw\" (UID: \"d95286cd-ebcd-47c2-9a95-d51f5e62576e\") " pod="openshift-image-registry/node-ca-2zpjw"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.300994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-registration-dir\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-run-ovn\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-node-log\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-multus-socket-dir-parent\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8-agent-certs\") pod \"konnectivity-agent-5vf27\" (UID: \"cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8\") " pod="kube-system/konnectivity-agent-5vf27"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-run-netns\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-run-multus-certs\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-run-netns\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-multus-socket-dir-parent\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.303491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-689xq\" (UniqueName: \"kubernetes.io/projected/87b01bad-5f40-4c30-aa70-bfd96d239873-kube-api-access-689xq\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.304156 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.304156 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-host-run-multus-certs\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.304156 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8-konnectivity-ca\") pod \"konnectivity-agent-5vf27\" (UID: \"cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8\") " pod="kube-system/konnectivity-agent-5vf27"
Sep 10 16:17:24.304156 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d3fba85-1986-4255-befb-90e94d9aa2e4-ovnkube-config\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.304156 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-system-cni-dir\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.304156 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f1f8695-2093-47a4-8397-730fcd19cedc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.304156 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.301453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-system-cni-dir\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.304494 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.304309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cc32931b-ee56-4382-96bb-2bbb378d6115-etc-tuned\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.305175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.305149    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc32931b-ee56-4382-96bb-2bbb378d6115-tmp\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.307830 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.307802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tj95\" (UniqueName: \"kubernetes.io/projected/0ed41c9e-f856-44a3-b13d-b6944bbfa63b-kube-api-access-4tj95\") pod \"iptables-alerter-wdd9f\" (UID: \"0ed41c9e-f856-44a3-b13d-b6944bbfa63b\") " pod="openshift-network-operator/iptables-alerter-wdd9f"
Sep 10 16:17:24.308139 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.308117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qctr6\" (UniqueName: \"kubernetes.io/projected/d95286cd-ebcd-47c2-9a95-d51f5e62576e-kube-api-access-qctr6\") pod \"node-ca-2zpjw\" (UID: \"d95286cd-ebcd-47c2-9a95-d51f5e62576e\") " pod="openshift-image-registry/node-ca-2zpjw"
Sep 10 16:17:24.309978 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.309953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-689xq\" (UniqueName: \"kubernetes.io/projected/87b01bad-5f40-4c30-aa70-bfd96d239873-kube-api-access-689xq\") pod \"multus-additional-cni-plugins-t4wn7\" (UID: \"87b01bad-5f40-4c30-aa70-bfd96d239873\") " pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.310123 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.310101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srqz4\" (UniqueName: \"kubernetes.io/projected/5f1f8695-2093-47a4-8397-730fcd19cedc-kube-api-access-srqz4\") pod \"aws-ebs-csi-driver-node-5l89w\" (UID: \"5f1f8695-2093-47a4-8397-730fcd19cedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.311026 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.310462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppcdb\" (UniqueName: \"kubernetes.io/projected/cc32931b-ee56-4382-96bb-2bbb378d6115-kube-api-access-ppcdb\") pod \"tuned-95jqx\" (UID: \"cc32931b-ee56-4382-96bb-2bbb378d6115\") " pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.311026 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.310517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb55l\" (UniqueName: \"kubernetes.io/projected/2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8-kube-api-access-zb55l\") pod \"node-resolver-pgclc\" (UID: \"2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8\") " pod="openshift-dns/node-resolver-pgclc"
Sep 10 16:17:24.311473 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.311453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmjq\" (UniqueName: \"kubernetes.io/projected/f719995b-ab3b-4e8c-a7b0-d055cd79caf5-kube-api-access-jvmjq\") pod \"multus-72nfz\" (UID: \"f719995b-ab3b-4e8c-a7b0-d055cd79caf5\") " pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.401684 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.401645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8-agent-certs\") pod \"konnectivity-agent-5vf27\" (UID: \"cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8\") " pod="kube-system/konnectivity-agent-5vf27"
Sep 10 16:17:24.401684 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.401689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8-konnectivity-ca\") pod \"konnectivity-agent-5vf27\" (UID: \"cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8\") " pod="kube-system/konnectivity-agent-5vf27"
Sep 10 16:17:24.401928 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.401714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d3fba85-1986-4255-befb-90e94d9aa2e4-ovnkube-config\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.401928 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.401741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-log-socket\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.401928 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.401770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-cni-bin\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.401928 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.401797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-var-lib-openvswitch\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.401928 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.401844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25606248-47cf-4de0-aab7-48d1123e9901-dbus\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:24.401928 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.401867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d3fba85-1986-4255-befb-90e94d9aa2e4-env-overrides\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.401928 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.401902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6smd\" (UniqueName: \"kubernetes.io/projected/4d3fba85-1986-4255-befb-90e94d9aa2e4-kube-api-access-d6smd\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402290 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.401935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpwq\" (UniqueName: \"kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq\") pod \"network-check-target-vzlvs\" (UID: \"218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f\") " pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:24.402290 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.401961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25606248-47cf-4de0-aab7-48d1123e9901-kubelet-config\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:24.402290 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.401989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-run-systemd\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402290 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-etc-openvswitch\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402290 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-kubelet\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402290 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-cni-netd\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402290 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25606248-47cf-4de0-aab7-48d1123e9901-dbus\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:24.402290 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d3fba85-1986-4255-befb-90e94d9aa2e4-ovn-node-metrics-cert\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402290 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402290 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402290 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-systemd-units\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402290 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d3fba85-1986-4255-befb-90e94d9aa2e4-ovnkube-script-lib\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402290 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:24.402958 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-run-netns\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402958 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-run-openvswitch\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402958 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8-konnectivity-ca\") pod \"konnectivity-agent-5vf27\" (UID: \"cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8\") " pod="kube-system/konnectivity-agent-5vf27"
Sep 10 16:17:24.402958 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs\") pod \"network-metrics-daemon-lnx4z\" (UID: \"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10\") " pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:24.402958 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvb2q\" (UniqueName: \"kubernetes.io/projected/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-kube-api-access-gvb2q\") pod \"network-metrics-daemon-lnx4z\" (UID: \"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10\") " pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:24.402958 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-slash\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402958 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-cni-bin\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402958 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-run-ovn\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402958 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-node-log\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402958 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-log-socket\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.402958 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-node-log\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402578    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25606248-47cf-4de0-aab7-48d1123e9901-kubelet-config\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-etc-openvswitch\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-var-lib-openvswitch\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-run-systemd\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-run-netns\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-run-openvswitch\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.402800    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-kubelet\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-cni-netd\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402963    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-host-slash\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.402993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-run-ovn\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.403000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d3fba85-1986-4255-befb-90e94d9aa2e4-systemd-units\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.403203    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret podName:25606248-47cf-4de0-aab7-48d1123e9901 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:24.903180419 +0000 UTC m=+3.104222838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret") pod "global-pull-secret-syncer-dc7jq" (UID: "25606248-47cf-4de0-aab7-48d1123e9901") : object "kube-system"/"original-pull-secret" not registered
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.402873    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 16:17:24.403432 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.403250    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs podName:0b2115e2-15e0-4fe4-9bfd-1158f59a0c10 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:24.90323836 +0000 UTC m=+3.104280766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs") pod "network-metrics-daemon-lnx4z" (UID: "0b2115e2-15e0-4fe4-9bfd-1158f59a0c10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 16:17:24.404215 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.403786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d3fba85-1986-4255-befb-90e94d9aa2e4-ovnkube-config\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.404215 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.404113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d3fba85-1986-4255-befb-90e94d9aa2e4-env-overrides\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.404327 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.404309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d3fba85-1986-4255-befb-90e94d9aa2e4-ovnkube-script-lib\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.404797 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.404732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8-agent-certs\") pod \"konnectivity-agent-5vf27\" (UID: \"cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8\") " pod="kube-system/konnectivity-agent-5vf27"
Sep 10 16:17:24.406164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.406125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d3fba85-1986-4255-befb-90e94d9aa2e4-ovn-node-metrics-cert\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.413309 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.413281    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 16:17:24.413309 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.413311    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 16:17:24.413501 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.413325    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nvpwq for pod openshift-network-diagnostics/network-check-target-vzlvs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 16:17:24.413501 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.413399    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq podName:218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f nodeName:}" failed. No retries permitted until 2026-09-10 16:17:24.913378307 +0000 UTC m=+3.114420702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nvpwq" (UniqueName: "kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq") pod "network-check-target-vzlvs" (UID: "218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 16:17:24.416585 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.416509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvb2q\" (UniqueName: \"kubernetes.io/projected/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-kube-api-access-gvb2q\") pod \"network-metrics-daemon-lnx4z\" (UID: \"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10\") " pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:24.418346 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.418322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6smd\" (UniqueName: \"kubernetes.io/projected/4d3fba85-1986-4255-befb-90e94d9aa2e4-kube-api-access-d6smd\") pod \"ovnkube-node-mbjsw\" (UID: \"4d3fba85-1986-4255-befb-90e94d9aa2e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.487230 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.487154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-95jqx"
Sep 10 16:17:24.496576 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.496536    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-72nfz"
Sep 10 16:17:24.508716 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.508683    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wdd9f"
Sep 10 16:17:24.515842 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.515817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w"
Sep 10 16:17:24.523776 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.523748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pgclc"
Sep 10 16:17:24.532696 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.532666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2zpjw"
Sep 10 16:17:24.540676 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.540646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t4wn7"
Sep 10 16:17:24.549629 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.549602    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:24.556643 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.556617    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5vf27"
Sep 10 16:17:24.908114 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.908015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:24.908114 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:24.908062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs\") pod \"network-metrics-daemon-lnx4z\" (UID: \"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10\") " pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:24.908366 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.908195    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 16:17:24.908366 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.908222    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 16:17:24.908366 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.908277    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret podName:25606248-47cf-4de0-aab7-48d1123e9901 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:25.908258051 +0000 UTC m=+4.109300442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret") pod "global-pull-secret-syncer-dc7jq" (UID: "25606248-47cf-4de0-aab7-48d1123e9901") : object "kube-system"/"original-pull-secret" not registered
Sep 10 16:17:24.908366 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:24.908299    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs podName:0b2115e2-15e0-4fe4-9bfd-1158f59a0c10 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:25.908289012 +0000 UTC m=+4.109331409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs") pod "network-metrics-daemon-lnx4z" (UID: "0b2115e2-15e0-4fe4-9bfd-1158f59a0c10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 16:17:25.009040 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:25.008995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpwq\" (UniqueName: \"kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq\") pod \"network-check-target-vzlvs\" (UID: \"218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f\") " pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:25.009209 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:25.009180    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 16:17:25.009209 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:25.009205    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 16:17:25.009299 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:25.009216    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nvpwq for pod openshift-network-diagnostics/network-check-target-vzlvs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 16:17:25.009299 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:25.009283    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq podName:218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f nodeName:}" failed. No retries permitted until 2026-09-10 16:17:26.009267937 +0000 UTC m=+4.210310327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvpwq" (UniqueName: "kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq") pod "network-check-target-vzlvs" (UID: "218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 16:17:25.243456 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:25.243353    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 16:12:24 +0000 UTC" deadline="2028-03-15 01:31:30.662680008 +0000 UTC"
Sep 10 16:17:25.243456 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:25.243407    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13233h14m5.419276298s"
Sep 10 16:17:25.543263 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:25.543233    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5e8ba0_2f3b_49df_b7e0_221a9ff548b8.slice/crio-91f6add795947bad75edbe6726cee841fb43a8b2cd70c257e34adf71616c1bc9 WatchSource:0}: Error finding container 91f6add795947bad75edbe6726cee841fb43a8b2cd70c257e34adf71616c1bc9: Status 404 returned error can't find the container with id 91f6add795947bad75edbe6726cee841fb43a8b2cd70c257e34adf71616c1bc9
Sep 10 16:17:25.544668 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:25.544431    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1f8695_2093_47a4_8397_730fcd19cedc.slice/crio-6dcf79a889651b9b51819228ef61f873b98e99e579933a158de43c881af82922 WatchSource:0}: Error finding container 6dcf79a889651b9b51819228ef61f873b98e99e579933a158de43c881af82922: Status 404 returned error can't find the container with id 6dcf79a889651b9b51819228ef61f873b98e99e579933a158de43c881af82922
Sep 10 16:17:25.546534 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:25.546499    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc32931b_ee56_4382_96bb_2bbb378d6115.slice/crio-3cc5dd98c8888ae09de9e13b7e91f2f95b0cb4b26ce887d9b5931a83a79b0ed7 WatchSource:0}: Error finding container 3cc5dd98c8888ae09de9e13b7e91f2f95b0cb4b26ce887d9b5931a83a79b0ed7: Status 404 returned error can't find the container with id 3cc5dd98c8888ae09de9e13b7e91f2f95b0cb4b26ce887d9b5931a83a79b0ed7
Sep 10 16:17:25.550779 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:25.549691    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95286cd_ebcd_47c2_9a95_d51f5e62576e.slice/crio-180f07cf65a0e21bee493d7abb84b401e041b56d82780102f932739aca9b8312 WatchSource:0}: Error finding container 180f07cf65a0e21bee493d7abb84b401e041b56d82780102f932739aca9b8312: Status 404 returned error can't find the container with id 180f07cf65a0e21bee493d7abb84b401e041b56d82780102f932739aca9b8312
Sep 10 16:17:25.550779 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:25.550757    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf719995b_ab3b_4e8c_a7b0_d055cd79caf5.slice/crio-916c19048d88e1550ba179e93491f2bce25f8d439bf49647e4c572cbab53c936 WatchSource:0}: Error finding container 916c19048d88e1550ba179e93491f2bce25f8d439bf49647e4c572cbab53c936: Status 404 returned error can't find the container with id 916c19048d88e1550ba179e93491f2bce25f8d439bf49647e4c572cbab53c936
Sep 10 16:17:25.553174 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:25.553141    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3fba85_1986_4255_befb_90e94d9aa2e4.slice/crio-74d68777a7da74bcc71dd47859129389758f7d942ffa95ccbe9955c88a03e8f9 WatchSource:0}: Error finding container 74d68777a7da74bcc71dd47859129389758f7d942ffa95ccbe9955c88a03e8f9: Status 404 returned error can't find the container with id 74d68777a7da74bcc71dd47859129389758f7d942ffa95ccbe9955c88a03e8f9
Sep 10 16:17:25.554344 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:25.553888    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed41c9e_f856_44a3_b13d_b6944bbfa63b.slice/crio-83259f7b3a6d35ee06de0ddf380085f6f572d07143a2ff2f547701d2d7e05897 WatchSource:0}: Error finding container 83259f7b3a6d35ee06de0ddf380085f6f572d07143a2ff2f547701d2d7e05897: Status 404 returned error can't find the container with id 83259f7b3a6d35ee06de0ddf380085f6f572d07143a2ff2f547701d2d7e05897
Sep 10 16:17:25.555343 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:25.555045    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa5acfb_ee22_43bb_9ca6_64284bb4dcb8.slice/crio-6dabc345c588f0431c02a80636be6b0568c4dedbaa06e7c729b936b30218fcb0 WatchSource:0}: Error finding container 6dabc345c588f0431c02a80636be6b0568c4dedbaa06e7c729b936b30218fcb0: Status 404 returned error can't find the container with id 6dabc345c588f0431c02a80636be6b0568c4dedbaa06e7c729b936b30218fcb0
Sep 10 16:17:25.557438 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:17:25.557255    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b01bad_5f40_4c30_aa70_bfd96d239873.slice/crio-2c420469f8627eb7b56c61c5669892f51b6fc2903de49541ddbfe481365bd586 WatchSource:0}: Error finding container 2c420469f8627eb7b56c61c5669892f51b6fc2903de49541ddbfe481365bd586: Status 404 returned error can't find the container with id 2c420469f8627eb7b56c61c5669892f51b6fc2903de49541ddbfe481365bd586
Sep 10 16:17:25.916697 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:25.916670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:25.916838 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:25.916705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs\") pod \"network-metrics-daemon-lnx4z\" (UID: \"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10\") " pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:25.916838 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:25.916814    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 16:17:25.916838 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:25.916826    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 16:17:25.916959 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:25.916873    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs podName:0b2115e2-15e0-4fe4-9bfd-1158f59a0c10 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:27.916860203 +0000 UTC m=+6.117902593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs") pod "network-metrics-daemon-lnx4z" (UID: "0b2115e2-15e0-4fe4-9bfd-1158f59a0c10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 16:17:25.916959 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:25.916887    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret podName:25606248-47cf-4de0-aab7-48d1123e9901 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:27.916880902 +0000 UTC m=+6.117923291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret") pod "global-pull-secret-syncer-dc7jq" (UID: "25606248-47cf-4de0-aab7-48d1123e9901") : object "kube-system"/"original-pull-secret" not registered
Sep 10 16:17:26.018054 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.018029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpwq\" (UniqueName: \"kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq\") pod \"network-check-target-vzlvs\" (UID: \"218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f\") " pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:26.018191 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:26.018174    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 16:17:26.018235 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:26.018197    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 16:17:26.018235 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:26.018207    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nvpwq for pod openshift-network-diagnostics/network-check-target-vzlvs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 16:17:26.018301 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:26.018253    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq podName:218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f nodeName:}" failed. No retries permitted until 2026-09-10 16:17:28.018240246 +0000 UTC m=+6.219282642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvpwq" (UniqueName: "kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq") pod "network-check-target-vzlvs" (UID: "218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 16:17:26.243646 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.243525    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 16:12:24 +0000 UTC" deadline="2028-02-28 02:50:12.542528681 +0000 UTC"
Sep 10 16:17:26.243646 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.243582    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12850h32m46.298951442s"
Sep 10 16:17:26.251807 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.251782    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:26.251964 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:26.251912    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:26.252344 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.252324    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:26.252459 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:26.252417    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:26.252529 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.252485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:26.252595 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:26.252564    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:26.269363 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.269292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-95jqx" event={"ID":"cc32931b-ee56-4382-96bb-2bbb378d6115","Type":"ContainerStarted","Data":"3cc5dd98c8888ae09de9e13b7e91f2f95b0cb4b26ce887d9b5931a83a79b0ed7"}
Sep 10 16:17:26.274425 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.274334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w" event={"ID":"5f1f8695-2093-47a4-8397-730fcd19cedc","Type":"ContainerStarted","Data":"6dcf79a889651b9b51819228ef61f873b98e99e579933a158de43c881af82922"}
Sep 10 16:17:26.284373 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.284344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" event={"ID":"18ee5e215f5e8ba4676df80e0d119df6","Type":"ContainerStarted","Data":"8f1940673aa91d99bff7ee5d0930c2263fde74bd6b5c7117a736a28e120d20e9"}
Sep 10 16:17:26.288973 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.288939    2580 generic.go:356] "Generic (PLEG): container finished" podID="015c0be4345cf7cdba6b30c6e8b201df" containerID="86b897ebb43f26fbe3f8cf2076e1e60b43d17a18b1ca80e6639d5b689008a90d" exitCode=0
Sep 10 16:17:26.289072 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.289040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" event={"ID":"015c0be4345cf7cdba6b30c6e8b201df","Type":"ContainerDied","Data":"86b897ebb43f26fbe3f8cf2076e1e60b43d17a18b1ca80e6639d5b689008a90d"}
Sep 10 16:17:26.300318 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.300293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4wn7" event={"ID":"87b01bad-5f40-4c30-aa70-bfd96d239873","Type":"ContainerStarted","Data":"2c420469f8627eb7b56c61c5669892f51b6fc2903de49541ddbfe481365bd586"}
Sep 10 16:17:26.311023 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.311001    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pgclc" event={"ID":"2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8","Type":"ContainerStarted","Data":"6dabc345c588f0431c02a80636be6b0568c4dedbaa06e7c729b936b30218fcb0"}
Sep 10 16:17:26.317116 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.317065    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" podStartSLOduration=3.317046835 podStartE2EDuration="3.317046835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:26.302254338 +0000 UTC m=+4.503296753" watchObservedRunningTime="2026-09-10 16:17:26.317046835 +0000 UTC m=+4.518089247"
Sep 10 16:17:26.320751 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.320727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wdd9f" event={"ID":"0ed41c9e-f856-44a3-b13d-b6944bbfa63b","Type":"ContainerStarted","Data":"83259f7b3a6d35ee06de0ddf380085f6f572d07143a2ff2f547701d2d7e05897"}
Sep 10 16:17:26.333371 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.333344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw" event={"ID":"4d3fba85-1986-4255-befb-90e94d9aa2e4","Type":"ContainerStarted","Data":"74d68777a7da74bcc71dd47859129389758f7d942ffa95ccbe9955c88a03e8f9"}
Sep 10 16:17:26.349262 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.349238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-72nfz" event={"ID":"f719995b-ab3b-4e8c-a7b0-d055cd79caf5","Type":"ContainerStarted","Data":"916c19048d88e1550ba179e93491f2bce25f8d439bf49647e4c572cbab53c936"}
Sep 10 16:17:26.378200 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.378168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5vf27" event={"ID":"cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8","Type":"ContainerStarted","Data":"91f6add795947bad75edbe6726cee841fb43a8b2cd70c257e34adf71616c1bc9"}
Sep 10 16:17:26.403881 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:26.403850    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2zpjw" event={"ID":"d95286cd-ebcd-47c2-9a95-d51f5e62576e","Type":"ContainerStarted","Data":"180f07cf65a0e21bee493d7abb84b401e041b56d82780102f932739aca9b8312"}
Sep 10 16:17:27.426681 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:27.424521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" event={"ID":"015c0be4345cf7cdba6b30c6e8b201df","Type":"ContainerStarted","Data":"8d10c4e893edd37ddc94854133543b3dd9080bd18858eb2b2d2001cf03db3c88"}
Sep 10 16:17:27.934780 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:27.934181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:27.934780 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:27.934240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs\") pod \"network-metrics-daemon-lnx4z\" (UID: \"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10\") " pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:27.934780 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:27.934365    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 16:17:27.934780 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:27.934430    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs podName:0b2115e2-15e0-4fe4-9bfd-1158f59a0c10 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:31.934410737 +0000 UTC m=+10.135453131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs") pod "network-metrics-daemon-lnx4z" (UID: "0b2115e2-15e0-4fe4-9bfd-1158f59a0c10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 16:17:27.934780 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:27.934366    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 16:17:27.934780 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:27.934518    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret podName:25606248-47cf-4de0-aab7-48d1123e9901 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:31.934499707 +0000 UTC m=+10.135542104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret") pod "global-pull-secret-syncer-dc7jq" (UID: "25606248-47cf-4de0-aab7-48d1123e9901") : object "kube-system"/"original-pull-secret" not registered
Sep 10 16:17:28.034922 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:28.034747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpwq\" (UniqueName: \"kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq\") pod \"network-check-target-vzlvs\" (UID: \"218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f\") " pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:28.035172 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:28.034984    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 16:17:28.035172 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:28.035005    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 16:17:28.035172 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:28.035019    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nvpwq for pod openshift-network-diagnostics/network-check-target-vzlvs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 16:17:28.035172 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:28.035084    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq podName:218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f nodeName:}" failed. No retries permitted until 2026-09-10 16:17:32.035062867 +0000 UTC m=+10.236105280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvpwq" (UniqueName: "kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq") pod "network-check-target-vzlvs" (UID: "218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 16:17:28.249109 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:28.249018    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:28.249284 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:28.249174    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:28.249654 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:28.249632    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:28.249785 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:28.249742    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:28.249859 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:28.249813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:28.249920 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:28.249891    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:30.254776 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:30.254737    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:30.255463 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:30.254871    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:30.255463 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:30.255278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:30.255463 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:30.255397    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:30.255463 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:30.255440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:30.255689 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:30.255514    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:31.967452 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:31.967365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:31.967918 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:31.967484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs\") pod \"network-metrics-daemon-lnx4z\" (UID: \"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10\") " pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:31.967918 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:31.967526    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 16:17:31.967918 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:31.967582    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 16:17:31.967918 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:31.967626    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret podName:25606248-47cf-4de0-aab7-48d1123e9901 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:39.967604228 +0000 UTC m=+18.168646641 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret") pod "global-pull-secret-syncer-dc7jq" (UID: "25606248-47cf-4de0-aab7-48d1123e9901") : object "kube-system"/"original-pull-secret" not registered
Sep 10 16:17:31.967918 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:31.967645    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs podName:0b2115e2-15e0-4fe4-9bfd-1158f59a0c10 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:39.9676376 +0000 UTC m=+18.168679989 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs") pod "network-metrics-daemon-lnx4z" (UID: "0b2115e2-15e0-4fe4-9bfd-1158f59a0c10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 16:17:32.068202 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:32.068155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpwq\" (UniqueName: \"kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq\") pod \"network-check-target-vzlvs\" (UID: \"218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f\") " pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:32.068397 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:32.068382    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 16:17:32.068474 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:32.068408    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 16:17:32.068474 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:32.068422    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nvpwq for pod openshift-network-diagnostics/network-check-target-vzlvs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 16:17:32.068597 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:32.068481    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq podName:218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f nodeName:}" failed. No retries permitted until 2026-09-10 16:17:40.068467529 +0000 UTC m=+18.269509920 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvpwq" (UniqueName: "kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq") pod "network-check-target-vzlvs" (UID: "218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 16:17:32.249421 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:32.249327    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:32.249587 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:32.249425    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:32.249587 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:32.249437    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:32.249712 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:32.249581    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:32.249712 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:32.249649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:32.249815 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:32.249726    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:34.252129 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:34.252052    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:34.252580 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:34.252054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:34.252580 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:34.252148    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:34.252580 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:34.252059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:34.252580 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:34.252267    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:34.252580 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:34.252345    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:36.251615 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:36.251579    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:36.251615 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:36.251607    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:36.252039 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:36.251607    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:36.252039 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:36.251717    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:36.252039 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:36.251830    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:36.252039 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:36.251880    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:38.252202 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:38.252172    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:38.252666 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:38.252277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:38.252666 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:38.252288    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:38.252666 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:38.252389    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:38.252666 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:38.252433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:38.252666 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:38.252486    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:40.026583 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:40.026548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:40.026583 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:40.026590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs\") pod \"network-metrics-daemon-lnx4z\" (UID: \"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10\") " pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:40.027079 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:40.026710    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 16:17:40.027079 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:40.026731    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 16:17:40.027079 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:40.026785    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret podName:25606248-47cf-4de0-aab7-48d1123e9901 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:56.026771363 +0000 UTC m=+34.227813752 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret") pod "global-pull-secret-syncer-dc7jq" (UID: "25606248-47cf-4de0-aab7-48d1123e9901") : object "kube-system"/"original-pull-secret" not registered
Sep 10 16:17:40.027079 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:40.026807    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs podName:0b2115e2-15e0-4fe4-9bfd-1158f59a0c10 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:56.026797468 +0000 UTC m=+34.227839871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs") pod "network-metrics-daemon-lnx4z" (UID: "0b2115e2-15e0-4fe4-9bfd-1158f59a0c10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 16:17:40.127394 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:40.127360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpwq\" (UniqueName: \"kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq\") pod \"network-check-target-vzlvs\" (UID: \"218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f\") " pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:40.127563 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:40.127511    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 16:17:40.127563 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:40.127547    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 16:17:40.127563 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:40.127558    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nvpwq for pod openshift-network-diagnostics/network-check-target-vzlvs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 16:17:40.127745 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:40.127618    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq podName:218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f nodeName:}" failed. No retries permitted until 2026-09-10 16:17:56.127599882 +0000 UTC m=+34.328642271 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvpwq" (UniqueName: "kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq") pod "network-check-target-vzlvs" (UID: "218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 16:17:40.248427 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:40.248394    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:40.248618 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:40.248440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:40.248618 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:40.248397    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:40.248618 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:40.248515    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:40.248777 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:40.248639    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:40.248777 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:40.248730    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:42.249943 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:42.249904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:42.249943 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:42.249933    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:42.250427 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:42.249907    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:42.250427 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:42.250016    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:42.250427 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:42.250099    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:42.250427 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:42.250162    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:44.249015 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.248699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:44.249510 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.248700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:44.249510 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:44.249110    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:44.249510 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.248751    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:44.249510 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:44.249189    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:44.249510 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:44.249238    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:44.465623 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.465527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pgclc" event={"ID":"2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8","Type":"ContainerStarted","Data":"766adea4ad9251f708ace6ea72e1635560b9be36edbf1127419433aa411e1674"}
Sep 10 16:17:44.468506 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.468480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw" event={"ID":"4d3fba85-1986-4255-befb-90e94d9aa2e4","Type":"ContainerStarted","Data":"0326a7fcf47b2de94d9fc0cc2860eaa796fb98280d8a61fff26ce27dca45ad1a"}
Sep 10 16:17:44.468664 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.468587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw" event={"ID":"4d3fba85-1986-4255-befb-90e94d9aa2e4","Type":"ContainerStarted","Data":"28f8954fcef92f983e04be8025b93bae2ce227ab31f23903b14a001635b89a96"}
Sep 10 16:17:44.468664 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.468641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw" event={"ID":"4d3fba85-1986-4255-befb-90e94d9aa2e4","Type":"ContainerStarted","Data":"d5d9b0cc125958cc2eb9a577ab9d162fd5d91010c95f49a87f3fe6983f4ee06b"}
Sep 10 16:17:44.468765 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.468690    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw" event={"ID":"4d3fba85-1986-4255-befb-90e94d9aa2e4","Type":"ContainerStarted","Data":"d2d001d286369f5db549c226f05aa9952a5ab60814b8b2eb89127ea741e7aecf"}
Sep 10 16:17:44.468765 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.468733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw" event={"ID":"4d3fba85-1986-4255-befb-90e94d9aa2e4","Type":"ContainerStarted","Data":"f987ba41fad2db94a6fd9bede4d2ba79c35f7f5b5496dfc92c6711ffb32052c2"}
Sep 10 16:17:44.468842 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.468770    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw" event={"ID":"4d3fba85-1986-4255-befb-90e94d9aa2e4","Type":"ContainerStarted","Data":"354488429b9631c6875eb9096217fdbaa4875e430d7d7ef57a97f207128bad94"}
Sep 10 16:17:44.469974 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.469953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-72nfz" event={"ID":"f719995b-ab3b-4e8c-a7b0-d055cd79caf5","Type":"ContainerStarted","Data":"9a268bf5c143e30e77b4d805c3df6f93100c01b5f3d93d57160463175bae8243"}
Sep 10 16:17:44.471390 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.471359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5vf27" event={"ID":"cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8","Type":"ContainerStarted","Data":"9943b87debb2d18941aabf034e7f731cda3e87a70c920cd3083629e06c5b0b68"}
Sep 10 16:17:44.472687 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.472666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2zpjw" event={"ID":"d95286cd-ebcd-47c2-9a95-d51f5e62576e","Type":"ContainerStarted","Data":"a2be07795cc6b7fcf30c0f62b4255b81f82406283ddc525983e98c1aa618ac93"}
Sep 10 16:17:44.473970 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.473950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-95jqx" event={"ID":"cc32931b-ee56-4382-96bb-2bbb378d6115","Type":"ContainerStarted","Data":"5facbf225fd4f18624d3feb6d1a3990352370c9ff5a96b4b7f159a77fe1e029a"}
Sep 10 16:17:44.475268 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.475249    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w" event={"ID":"5f1f8695-2093-47a4-8397-730fcd19cedc","Type":"ContainerStarted","Data":"ac297e7429634169889eed022ee98af4485fd701482dac70814f40afb4d9c1ce"}
Sep 10 16:17:44.476714 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.476691    2580 generic.go:356] "Generic (PLEG): container finished" podID="87b01bad-5f40-4c30-aa70-bfd96d239873" containerID="2dbf76f9d8e2c111c81bde95775eb6c456c47b7a005e8c70c85c8628436506e6" exitCode=0
Sep 10 16:17:44.476792 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.476721    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4wn7" event={"ID":"87b01bad-5f40-4c30-aa70-bfd96d239873","Type":"ContainerDied","Data":"2dbf76f9d8e2c111c81bde95775eb6c456c47b7a005e8c70c85c8628436506e6"}
Sep 10 16:17:44.484005 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.483966    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" podStartSLOduration=21.483956187 podStartE2EDuration="21.483956187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:27.443111843 +0000 UTC m=+5.644154258" watchObservedRunningTime="2026-09-10 16:17:44.483956187 +0000 UTC m=+22.684998594"
Sep 10 16:17:44.484187 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.484169    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pgclc" podStartSLOduration=3.365153446 podStartE2EDuration="21.484165693s" totalImagesPullingTime="18.119012247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:44.483777163 +0000 UTC m=+22.684819575" watchObservedRunningTime="2026-09-10 16:17:44.484165693 +0000 UTC m=+22.685208101"
Sep 10 16:17:44.498876 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.498842    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5vf27" podStartSLOduration=10.600438018 podStartE2EDuration="21.498830845s" totalImagesPullingTime="10.898392827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:44.498818283 +0000 UTC m=+22.699860691" watchObservedRunningTime="2026-09-10 16:17:44.498830845 +0000 UTC m=+22.699873250"
Sep 10 16:17:44.515228 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.515198    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-72nfz" podStartSLOduration=3.331211826 podStartE2EDuration="21.515182916s" totalImagesPullingTime="18.18397109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:44.514686894 +0000 UTC m=+22.715729305" watchObservedRunningTime="2026-09-10 16:17:44.515182916 +0000 UTC m=+22.716225394"
Sep 10 16:17:44.529493 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.529461    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2zpjw" podStartSLOduration=3.4048134340000002 podStartE2EDuration="21.529449955s" totalImagesPullingTime="18.124636521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:44.529297047 +0000 UTC m=+22.730339500" watchObservedRunningTime="2026-09-10 16:17:44.529449955 +0000 UTC m=+22.730492362"
Sep 10 16:17:44.570068 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.570034    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-95jqx" podStartSLOduration=3.441469074 podStartE2EDuration="21.57002349s" totalImagesPullingTime="18.128554416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:44.569632988 +0000 UTC m=+22.770675399" watchObservedRunningTime="2026-09-10 16:17:44.57002349 +0000 UTC m=+22.771065901"
Sep 10 16:17:44.663699 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.663665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5vf27_cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8/konnectivity-agent/0.log" limit=1024
Sep 10 16:17:44.743187 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:44.743133    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5vf27_cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8/konnectivity-agent/0.log" limit=1024
Sep 10 16:17:45.382154 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:45.382127    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 16:17:45.480513 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:45.480449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w" event={"ID":"5f1f8695-2093-47a4-8397-730fcd19cedc","Type":"ContainerStarted","Data":"539a2e3b806a54e50de097020c156948498e6e890de31b9feb5f43a9d55e9f8d"}
Sep 10 16:17:45.481879 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:45.481836    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wdd9f" event={"ID":"0ed41c9e-f856-44a3-b13d-b6944bbfa63b","Type":"ContainerStarted","Data":"bc76362a1536a78527258147af2b7403a59cc6e94972f7a20607502596059bb3"}
Sep 10 16:17:45.493165 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:45.493143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5vf27_cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8/konnectivity-agent/0.log" limit=1024
Sep 10 16:17:45.597651 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:45.597624    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5vf27_cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8/konnectivity-agent/0.log" limit=1024
Sep 10 16:17:45.694170 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:45.694134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5vf27_cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8/konnectivity-agent/0.log" limit=1024
Sep 10 16:17:45.869628 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:45.869551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5vf27_cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8/konnectivity-agent/0.log" limit=1024
Sep 10 16:17:45.979165 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:45.979129    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5vf27_cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8/konnectivity-agent/0.log" limit=1024
Sep 10 16:17:46.044758 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:46.044728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5vf27_cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8/konnectivity-agent/0.log" limit=1024
Sep 10 16:17:46.247936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:46.247777    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T16:17:45.382146255Z","UUID":"c86518b9-946d-4373-beaf-d3f865933716","Handler":null,"Name":"","Endpoint":""}
Sep 10 16:17:46.248359 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:46.248335    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:46.248476 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:46.248457    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:46.248608 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:46.248583    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:46.249123 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:46.248902    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:46.249123 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:46.249010    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:46.249123 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:46.249096    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:46.249899 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:46.249878    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 16:17:46.250003 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:46.249915    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 16:17:46.486312 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:46.486014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w" event={"ID":"5f1f8695-2093-47a4-8397-730fcd19cedc","Type":"ContainerStarted","Data":"51d4cb46d773e5206c79f536c89700b2185239c30cc47d99466e16d8637438ee"}
Sep 10 16:17:46.489198 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:46.489171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw" event={"ID":"4d3fba85-1986-4255-befb-90e94d9aa2e4","Type":"ContainerStarted","Data":"bd05893d2df5d8088c34bd564d525fd420b6ac5036f0af5b8675df4179be9b82"}
Sep 10 16:17:46.504120 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:46.504082    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wdd9f" podStartSLOduration=5.38407709 podStartE2EDuration="23.504070264s" totalImagesPullingTime="18.119993174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:45.500421038 +0000 UTC m=+23.701463453" watchObservedRunningTime="2026-09-10 16:17:46.504070264 +0000 UTC m=+24.705112674"
Sep 10 16:17:46.504229 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:46.504175    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5l89w" podStartSLOduration=3.196900891 podStartE2EDuration="23.504169821s" totalImagesPullingTime="20.30726893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:46.503938409 +0000 UTC m=+24.704980825" watchObservedRunningTime="2026-09-10 16:17:46.504169821 +0000 UTC m=+24.705212233"
Sep 10 16:17:46.908175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:46.908107    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5vf27_cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8/konnectivity-agent/0.log" limit=1024
Sep 10 16:17:46.967987 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:46.967937    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5vf27_cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8/konnectivity-agent/0.log" limit=1024
Sep 10 16:17:48.248948 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:48.248916    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:48.249390 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:48.248921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:48.249390 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:48.249039    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:48.249390 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:48.249062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:48.249390 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:48.249139    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:48.249390 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:48.249237    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:48.595869 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:48.595786    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5vf27"
Sep 10 16:17:48.596547 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:48.596507    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5vf27"
Sep 10 16:17:50.248709 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:50.248482    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:50.249108 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:50.248482    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:50.249108 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:50.248758    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:50.249108 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:50.248859    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:50.249108 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:50.248488    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:50.249108 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:50.248967    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:50.498964 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:50.498906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4wn7" event={"ID":"87b01bad-5f40-4c30-aa70-bfd96d239873","Type":"ContainerStarted","Data":"130161eb9ad9921f8de0b7c335a3830ebc1293827768e3dc6a5187bf5907b5f3"}
Sep 10 16:17:50.502211 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:50.502186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw" event={"ID":"4d3fba85-1986-4255-befb-90e94d9aa2e4","Type":"ContainerStarted","Data":"ee71fe4242c2392022267c9fa6cfb87c4ec8dad89705786d330d734ef5d0ce10"}
Sep 10 16:17:50.502410 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:50.502391    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:50.502475 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:50.502468    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:50.516850 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:50.516828    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:50.545839 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:50.545804    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw" podStartSLOduration=9.36412532 podStartE2EDuration="27.545791307s" totalImagesPullingTime="18.181665987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:50.544888817 +0000 UTC m=+28.745931227" watchObservedRunningTime="2026-09-10 16:17:50.545791307 +0000 UTC m=+28.746833717"
Sep 10 16:17:51.506718 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:51.506686    2580 generic.go:356] "Generic (PLEG): container finished" podID="87b01bad-5f40-4c30-aa70-bfd96d239873" containerID="130161eb9ad9921f8de0b7c335a3830ebc1293827768e3dc6a5187bf5907b5f3" exitCode=0
Sep 10 16:17:51.507604 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:51.506769    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4wn7" event={"ID":"87b01bad-5f40-4c30-aa70-bfd96d239873","Type":"ContainerDied","Data":"130161eb9ad9921f8de0b7c335a3830ebc1293827768e3dc6a5187bf5907b5f3"}
Sep 10 16:17:51.507604 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:51.507133    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:51.521588 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:51.521568    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:17:51.862714 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:51.862645    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vzlvs"]
Sep 10 16:17:51.862851 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:51.862770    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:51.862922 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:51.862855    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:51.866664 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:51.866639    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lnx4z"]
Sep 10 16:17:51.866776 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:51.866743    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:51.866874 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:51.866851    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:51.867207 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:51.867184    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dc7jq"]
Sep 10 16:17:51.867286 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:51.867271    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:51.867363 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:51.867344    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:52.788738 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:52.788480    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5vf27"
Sep 10 16:17:52.789195 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:52.788783    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 16:17:52.789195 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:52.789002    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5vf27"
Sep 10 16:17:53.248686 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:53.248656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:53.248831 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:53.248757    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:53.248831 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:53.248808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:53.248924 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:53.248906    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:53.513160 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:53.513080    2580 generic.go:356] "Generic (PLEG): container finished" podID="87b01bad-5f40-4c30-aa70-bfd96d239873" containerID="779f837200347007b22473f2962b1d8098a1a3b3b30d437d44a88357844df3ce" exitCode=0
Sep 10 16:17:53.513160 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:53.513132    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4wn7" event={"ID":"87b01bad-5f40-4c30-aa70-bfd96d239873","Type":"ContainerDied","Data":"779f837200347007b22473f2962b1d8098a1a3b3b30d437d44a88357844df3ce"}
Sep 10 16:17:53.530825 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:53.530808    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pgclc_2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8/dns-node-resolver/0.log"
Sep 10 16:17:54.249111 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:54.249082    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:54.249478 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:54.249186    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:54.330425 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:54.330362    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2zpjw_d95286cd-ebcd-47c2-9a95-d51f5e62576e/node-ca/0.log"
Sep 10 16:17:55.248421 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:55.248395    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:55.248575 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:55.248398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:55.248575 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:55.248484    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dc7jq" podUID="25606248-47cf-4de0-aab7-48d1123e9901"
Sep 10 16:17:55.248649 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:55.248597    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnx4z" podUID="0b2115e2-15e0-4fe4-9bfd-1158f59a0c10"
Sep 10 16:17:55.519158 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:55.519091    2580 generic.go:356] "Generic (PLEG): container finished" podID="87b01bad-5f40-4c30-aa70-bfd96d239873" containerID="c81bc4df9a37011ff1d65fa7b6cd7c27a2ddd2be2bd22a4f6d19eb042908996b" exitCode=0
Sep 10 16:17:55.519158 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:55.519126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4wn7" event={"ID":"87b01bad-5f40-4c30-aa70-bfd96d239873","Type":"ContainerDied","Data":"c81bc4df9a37011ff1d65fa7b6cd7c27a2ddd2be2bd22a4f6d19eb042908996b"}
Sep 10 16:17:56.045098 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:56.045074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:56.045246 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:56.045105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs\") pod \"network-metrics-daemon-lnx4z\" (UID: \"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10\") " pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:56.045246 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:56.045208    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 16:17:56.045335 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:56.045258    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs podName:0b2115e2-15e0-4fe4-9bfd-1158f59a0c10 nodeName:}" failed. No retries permitted until 2026-09-10 16:18:28.045235525 +0000 UTC m=+66.246277915 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs") pod "network-metrics-daemon-lnx4z" (UID: "0b2115e2-15e0-4fe4-9bfd-1158f59a0c10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 16:17:56.045335 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:56.045207    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 16:17:56.045400 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:56.045334    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret podName:25606248-47cf-4de0-aab7-48d1123e9901 nodeName:}" failed. No retries permitted until 2026-09-10 16:18:28.045318181 +0000 UTC m=+66.246360571 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret") pod "global-pull-secret-syncer-dc7jq" (UID: "25606248-47cf-4de0-aab7-48d1123e9901") : object "kube-system"/"original-pull-secret" not registered
Sep 10 16:17:56.145814 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:56.145779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpwq\" (UniqueName: \"kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq\") pod \"network-check-target-vzlvs\" (UID: \"218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f\") " pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:56.145972 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:56.145952    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 16:17:56.145972 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:56.145972    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 16:17:56.146060 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:56.145983    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nvpwq for pod openshift-network-diagnostics/network-check-target-vzlvs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 16:17:56.146060 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:56.146033    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq podName:218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f nodeName:}" failed. No retries permitted until 2026-09-10 16:18:28.146014578 +0000 UTC m=+66.347056972 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvpwq" (UniqueName: "kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq") pod "network-check-target-vzlvs" (UID: "218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 16:17:56.249174 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:56.249151    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:56.249278 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:17:56.249242    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzlvs" podUID="218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f"
Sep 10 16:17:56.653205 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:56.653174    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeReady"
Sep 10 16:17:56.653653 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:56.653306    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 16:17:57.248301 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:57.248273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:17:57.248527 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:57.248433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:17:57.251032 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:57.251004    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j8r7d\""
Sep 10 16:17:57.251032 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:57.251028    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 16:17:57.251207 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:57.251038    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 16:17:58.249175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:58.249022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:17:58.251735 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:58.251716    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 16:17:58.252408 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:58.252379    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 16:17:58.252545 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:17:58.252380    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-htvxx\""
Sep 10 16:18:00.533318 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:00.533126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4wn7" event={"ID":"87b01bad-5f40-4c30-aa70-bfd96d239873","Type":"ContainerStarted","Data":"be019d7ed103ffa149a9f0dba123d1dded8d427b8fb37d2240e09ba6f9ec0511"}
Sep 10 16:18:01.537713 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:01.537683    2580 generic.go:356] "Generic (PLEG): container finished" podID="87b01bad-5f40-4c30-aa70-bfd96d239873" containerID="be019d7ed103ffa149a9f0dba123d1dded8d427b8fb37d2240e09ba6f9ec0511" exitCode=0
Sep 10 16:18:01.538065 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:01.537719    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4wn7" event={"ID":"87b01bad-5f40-4c30-aa70-bfd96d239873","Type":"ContainerDied","Data":"be019d7ed103ffa149a9f0dba123d1dded8d427b8fb37d2240e09ba6f9ec0511"}
Sep 10 16:18:02.542767 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:02.542733    2580 generic.go:356] "Generic (PLEG): container finished" podID="87b01bad-5f40-4c30-aa70-bfd96d239873" containerID="8941a2e18a05114b6f24368c23127419c4ced773e9dc751d4b361cb788466dfc" exitCode=0
Sep 10 16:18:02.543107 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:02.542790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4wn7" event={"ID":"87b01bad-5f40-4c30-aa70-bfd96d239873","Type":"ContainerDied","Data":"8941a2e18a05114b6f24368c23127419c4ced773e9dc751d4b361cb788466dfc"}
Sep 10 16:18:03.548591 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:03.548552    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4wn7" event={"ID":"87b01bad-5f40-4c30-aa70-bfd96d239873","Type":"ContainerStarted","Data":"715669b2acdae9e1afac0031b85922c858a8239776cb659204e4db887547338f"}
Sep 10 16:18:03.575752 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:03.575699    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t4wn7" podStartSLOduration=9.790795896 podStartE2EDuration="40.575681409s" totalImagesPullingTime="30.784885513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:03.574920611 +0000 UTC m=+41.775963035" watchObservedRunningTime="2026-09-10 16:18:03.575681409 +0000 UTC m=+41.776723822"
Sep 10 16:18:18.808938 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.808903    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79d99c44cf-6kjvv"]
Sep 10 16:18:18.810982 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.810960    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.813084 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.813049    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 16:18:18.813084 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.813088    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s2v9s\""
Sep 10 16:18:18.813254 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.813094    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 16:18:18.813254 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.813147    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 16:18:18.818819 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.818801    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 16:18:18.823483 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.823465    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79d99c44cf-6kjvv"]
Sep 10 16:18:18.887177 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.887150    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2bbf6"]
Sep 10 16:18:18.888722 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.888010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-registry-tls\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.888722 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.888061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-installation-pull-secrets\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.888722 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.888139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-bound-sa-token\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.888722 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.888191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qccx7\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-kube-api-access-qccx7\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.888722 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.888222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-image-registry-private-configuration\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.888722 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.888253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-registry-certificates\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.888722 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.888281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-trusted-ca\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.888722 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.888346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-ca-trust-extracted\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.889878 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.889857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2bbf6"
Sep 10 16:18:18.892810 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.892789    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 16:18:18.893213 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.893196    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 16:18:18.893708 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.893686    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kr9bx\""
Sep 10 16:18:18.893795 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.893715    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 16:18:18.902847 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.902828    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2bbf6"]
Sep 10 16:18:18.989040 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.989018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-bound-sa-token\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.989132 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.989049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmn5\" (UniqueName: \"kubernetes.io/projected/fc083413-78ac-40c4-b86c-16a0aec316f2-kube-api-access-6zmn5\") pod \"ingress-canary-2bbf6\" (UID: \"fc083413-78ac-40c4-b86c-16a0aec316f2\") " pod="openshift-ingress-canary/ingress-canary-2bbf6"
Sep 10 16:18:18.989132 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.989082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qccx7\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-kube-api-access-qccx7\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.989132 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.989112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-image-registry-private-configuration\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.989132 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.989131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-registry-certificates\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.989307 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.989145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-trusted-ca\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.989307 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.989186    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc083413-78ac-40c4-b86c-16a0aec316f2-cert\") pod \"ingress-canary-2bbf6\" (UID: \"fc083413-78ac-40c4-b86c-16a0aec316f2\") " pod="openshift-ingress-canary/ingress-canary-2bbf6"
Sep 10 16:18:18.989307 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.989220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-ca-trust-extracted\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.989307 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.989288    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-registry-tls\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.989517 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.989317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-installation-pull-secrets\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.989618 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.989599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-ca-trust-extracted\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.989980 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.989960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-registry-certificates\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.990206 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.990186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-trusted-ca\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.993033 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.993011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-installation-pull-secrets\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.993131 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.993097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-image-registry-private-configuration\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.993279 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.993261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-registry-tls\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:18.998494 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:18.998475    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ptvp7"]
Sep 10 16:18:19.000847 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.000831    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:19.001775 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.001752    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-m5m78"]
Sep 10 16:18:19.003369 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.003351    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-r2g6m"]
Sep 10 16:18:19.003482 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.003471    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-m5m78"
Sep 10 16:18:19.004903 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.004883    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wwcbb\""
Sep 10 16:18:19.005128 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.005110    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"]
Sep 10 16:18:19.005234 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.005209    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-r2g6m"
Sep 10 16:18:19.005376 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.005360    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 16:18:19.005690 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.005664    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 16:18:19.006911 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.006895    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k4nxn"]
Sep 10 16:18:19.007011 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.006997    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.008775 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.008755    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-865855d986-2dhqm"]
Sep 10 16:18:19.009055 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.009034    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.010313 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.010298    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"]
Sep 10 16:18:19.010438 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.010387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-865855d986-2dhqm"
Sep 10 16:18:19.011769 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.011752    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"
Sep 10 16:18:19.014223 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.014203    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 16:18:19.014223 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.014216    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 16:18:19.014361 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.014203    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kttk5\""
Sep 10 16:18:19.014361 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.014270    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ndzqx\""
Sep 10 16:18:19.015679 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.015663    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 10 16:18:19.015943 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.015930    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 16:18:19.016638 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.016619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qccx7\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-kube-api-access-qccx7\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:19.017522 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.017509    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n9wnm\""
Sep 10 16:18:19.017894 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.017867    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 16:18:19.018000 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.017982    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 16:18:19.022540 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.022513    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 16:18:19.023784 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.023761    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 16:18:19.024083 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.024064    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 16:18:19.024231 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.024119    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 16:18:19.024231 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.024149    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 16:18:19.024396 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.024239    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 16:18:19.025363 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.025346    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 16:18:19.025810 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.025792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-bound-sa-token\") pod \"image-registry-79d99c44cf-6kjvv\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") " pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:19.033436 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.033413    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 16:18:19.033518 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.033413    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 16:18:19.033719 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.033704    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9t8sf\""
Sep 10 16:18:19.033807 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.033792    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 16:18:19.050345 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.050324    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"]
Sep 10 16:18:19.051459 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.051441    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ptvp7"]
Sep 10 16:18:19.052881 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.052687    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"]
Sep 10 16:18:19.053796 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.053778    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-m5m78"]
Sep 10 16:18:19.054606 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.054583    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-865855d986-2dhqm"]
Sep 10 16:18:19.055589 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.055569    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79d99c44cf-6kjvv"]
Sep 10 16:18:19.055862 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.055830    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:19.060617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.060599    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k4nxn"]
Sep 10 16:18:19.069935 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.069911    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-r2g6m"]
Sep 10 16:18:19.089703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.089681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9cb81b47-97bd-4561-85c3-054dbcf875b5-nginx-conf\") pod \"networking-console-plugin-79977f9844-m5m78\" (UID: \"9cb81b47-97bd-4561-85c3-054dbcf875b5\") " pod="openshift-network-console/networking-console-plugin-79977f9844-m5m78"
Sep 10 16:18:19.089818 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.089713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8d04a4f4-b351-416a-9126-37c649ad0989-ca\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.089818 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.089736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n28xb\" (UniqueName: \"kubernetes.io/projected/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-kube-api-access-n28xb\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.089818 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.089774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.089818 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.089792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n785x\" (UniqueName: \"kubernetes.io/projected/3b5cd229-4191-436f-bae3-841284e8f8f2-kube-api-access-n785x\") pod \"dns-default-ptvp7\" (UID: \"3b5cd229-4191-436f-bae3-841284e8f8f2\") " pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:19.089818 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.089807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/604b4c6c-a072-450a-95df-091c974db590-klusterlet-config\") pod \"klusterlet-addon-workmgr-76658f465f-jpdxk\" (UID: \"604b4c6c-a072-450a-95df-091c974db590\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"
Sep 10 16:18:19.090017 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.089829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7pvb\" (UniqueName: \"kubernetes.io/projected/604b4c6c-a072-450a-95df-091c974db590-kube-api-access-s7pvb\") pod \"klusterlet-addon-workmgr-76658f465f-jpdxk\" (UID: \"604b4c6c-a072-450a-95df-091c974db590\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"
Sep 10 16:18:19.090017 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.089846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e5e1f877-d651-464b-9047-3c64e36a8f0d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-r2g6m\" (UID: \"e5e1f877-d651-464b-9047-3c64e36a8f0d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-r2g6m"
Sep 10 16:18:19.090017 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.089886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b5cd229-4191-436f-bae3-841284e8f8f2-metrics-tls\") pod \"dns-default-ptvp7\" (UID: \"3b5cd229-4191-436f-bae3-841284e8f8f2\") " pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:19.090017 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.089923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-crio-socket\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.090017 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.089964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmn5\" (UniqueName: \"kubernetes.io/projected/fc083413-78ac-40c4-b86c-16a0aec316f2-kube-api-access-6zmn5\") pod \"ingress-canary-2bbf6\" (UID: \"fc083413-78ac-40c4-b86c-16a0aec316f2\") " pod="openshift-ingress-canary/ingress-canary-2bbf6"
Sep 10 16:18:19.090017 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.089989    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2qz5\" (UniqueName: \"kubernetes.io/projected/8d04a4f4-b351-416a-9126-37c649ad0989-kube-api-access-c2qz5\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.090017 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.090006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d04a4f4-b351-416a-9126-37c649ad0989-additional-service-ca\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.090309 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.090022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5cd229-4191-436f-bae3-841284e8f8f2-tmp-dir\") pod \"dns-default-ptvp7\" (UID: \"3b5cd229-4191-436f-bae3-841284e8f8f2\") " pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:19.090309 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.090041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8d04a4f4-b351-416a-9126-37c649ad0989-hub\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.090309 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.090069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8d04a4f4-b351-416a-9126-37c649ad0989-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.090309 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.090086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc083413-78ac-40c4-b86c-16a0aec316f2-cert\") pod \"ingress-canary-2bbf6\" (UID: \"fc083413-78ac-40c4-b86c-16a0aec316f2\") " pod="openshift-ingress-canary/ingress-canary-2bbf6"
Sep 10 16:18:19.090309 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.090108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.090309 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.090153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5cd229-4191-436f-bae3-841284e8f8f2-config-volume\") pod \"dns-default-ptvp7\" (UID: \"3b5cd229-4191-436f-bae3-841284e8f8f2\") " pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:19.090309 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.090267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0033469b-a67d-401e-8a8d-7603387b30d6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-865855d986-2dhqm\" (UID: \"0033469b-a67d-401e-8a8d-7603387b30d6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-865855d986-2dhqm"
Sep 10 16:18:19.090585 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.090314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cb81b47-97bd-4561-85c3-054dbcf875b5-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-m5m78\" (UID: \"9cb81b47-97bd-4561-85c3-054dbcf875b5\") " pod="openshift-network-console/networking-console-plugin-79977f9844-m5m78"
Sep 10 16:18:19.090585 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.090351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdmmp\" (UniqueName: \"kubernetes.io/projected/0033469b-a67d-401e-8a8d-7603387b30d6-kube-api-access-vdmmp\") pod \"managed-serviceaccount-addon-agent-865855d986-2dhqm\" (UID: \"0033469b-a67d-401e-8a8d-7603387b30d6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-865855d986-2dhqm"
Sep 10 16:18:19.090585 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.090376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d04a4f4-b351-416a-9126-37c649ad0989-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.090585 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.090408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-data-volume\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.090585 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.090433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/604b4c6c-a072-450a-95df-091c974db590-tmp\") pod \"klusterlet-addon-workmgr-76658f465f-jpdxk\" (UID: \"604b4c6c-a072-450a-95df-091c974db590\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"
Sep 10 16:18:19.092892 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.092847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc083413-78ac-40c4-b86c-16a0aec316f2-cert\") pod \"ingress-canary-2bbf6\" (UID: \"fc083413-78ac-40c4-b86c-16a0aec316f2\") " pod="openshift-ingress-canary/ingress-canary-2bbf6"
Sep 10 16:18:19.111296 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.111279    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c6859976-hjrp4"]
Sep 10 16:18:19.114573 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.114553    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.133200 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.133182    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmn5\" (UniqueName: \"kubernetes.io/projected/fc083413-78ac-40c4-b86c-16a0aec316f2-kube-api-access-6zmn5\") pod \"ingress-canary-2bbf6\" (UID: \"fc083413-78ac-40c4-b86c-16a0aec316f2\") " pod="openshift-ingress-canary/ingress-canary-2bbf6"
Sep 10 16:18:19.153821 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.153799    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c6859976-hjrp4"]
Sep 10 16:18:19.191692 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.191667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2qz5\" (UniqueName: \"kubernetes.io/projected/8d04a4f4-b351-416a-9126-37c649ad0989-kube-api-access-c2qz5\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.191781 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.191696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d04a4f4-b351-416a-9126-37c649ad0989-additional-service-ca\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.191781 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.191715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5cd229-4191-436f-bae3-841284e8f8f2-tmp-dir\") pod \"dns-default-ptvp7\" (UID: \"3b5cd229-4191-436f-bae3-841284e8f8f2\") " pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:19.191781 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.191731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8d04a4f4-b351-416a-9126-37c649ad0989-hub\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.191781 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.191749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8d04a4f4-b351-416a-9126-37c649ad0989-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.191781 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.191772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.192002 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.191797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5cd229-4191-436f-bae3-841284e8f8f2-config-volume\") pod \"dns-default-ptvp7\" (UID: \"3b5cd229-4191-436f-bae3-841284e8f8f2\") " pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:19.192002 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.191821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0033469b-a67d-401e-8a8d-7603387b30d6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-865855d986-2dhqm\" (UID: \"0033469b-a67d-401e-8a8d-7603387b30d6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-865855d986-2dhqm"
Sep 10 16:18:19.192002 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.191849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cb81b47-97bd-4561-85c3-054dbcf875b5-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-m5m78\" (UID: \"9cb81b47-97bd-4561-85c3-054dbcf875b5\") " pod="openshift-network-console/networking-console-plugin-79977f9844-m5m78"
Sep 10 16:18:19.192152 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5cd229-4191-436f-bae3-841284e8f8f2-tmp-dir\") pod \"dns-default-ptvp7\" (UID: \"3b5cd229-4191-436f-bae3-841284e8f8f2\") " pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:19.192152 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdmmp\" (UniqueName: \"kubernetes.io/projected/0033469b-a67d-401e-8a8d-7603387b30d6-kube-api-access-vdmmp\") pod \"managed-serviceaccount-addon-agent-865855d986-2dhqm\" (UID: \"0033469b-a67d-401e-8a8d-7603387b30d6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-865855d986-2dhqm"
Sep 10 16:18:19.192152 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d04a4f4-b351-416a-9126-37c649ad0989-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.192301 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-data-volume\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.192301 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/604b4c6c-a072-450a-95df-091c974db590-tmp\") pod \"klusterlet-addon-workmgr-76658f465f-jpdxk\" (UID: \"604b4c6c-a072-450a-95df-091c974db590\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"
Sep 10 16:18:19.192301 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9cb81b47-97bd-4561-85c3-054dbcf875b5-nginx-conf\") pod \"networking-console-plugin-79977f9844-m5m78\" (UID: \"9cb81b47-97bd-4561-85c3-054dbcf875b5\") " pod="openshift-network-console/networking-console-plugin-79977f9844-m5m78"
Sep 10 16:18:19.192301 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8d04a4f4-b351-416a-9126-37c649ad0989-ca\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.192301 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n28xb\" (UniqueName: \"kubernetes.io/projected/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-kube-api-access-n28xb\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.192653 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.192653 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n785x\" (UniqueName: \"kubernetes.io/projected/3b5cd229-4191-436f-bae3-841284e8f8f2-kube-api-access-n785x\") pod \"dns-default-ptvp7\" (UID: \"3b5cd229-4191-436f-bae3-841284e8f8f2\") " pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:19.192653 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/604b4c6c-a072-450a-95df-091c974db590-klusterlet-config\") pod \"klusterlet-addon-workmgr-76658f465f-jpdxk\" (UID: \"604b4c6c-a072-450a-95df-091c974db590\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"
Sep 10 16:18:19.192653 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7pvb\" (UniqueName: \"kubernetes.io/projected/604b4c6c-a072-450a-95df-091c974db590-kube-api-access-s7pvb\") pod \"klusterlet-addon-workmgr-76658f465f-jpdxk\" (UID: \"604b4c6c-a072-450a-95df-091c974db590\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"
Sep 10 16:18:19.192653 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e5e1f877-d651-464b-9047-3c64e36a8f0d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-r2g6m\" (UID: \"e5e1f877-d651-464b-9047-3c64e36a8f0d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-r2g6m"
Sep 10 16:18:19.192653 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d04a4f4-b351-416a-9126-37c649ad0989-additional-service-ca\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.192653 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b5cd229-4191-436f-bae3-841284e8f8f2-metrics-tls\") pod \"dns-default-ptvp7\" (UID: \"3b5cd229-4191-436f-bae3-841284e8f8f2\") " pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:19.192653 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-crio-socket\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.193125 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-crio-socket\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.193125 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/604b4c6c-a072-450a-95df-091c974db590-tmp\") pod \"klusterlet-addon-workmgr-76658f465f-jpdxk\" (UID: \"604b4c6c-a072-450a-95df-091c974db590\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"
Sep 10 16:18:19.193125 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.192950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-data-volume\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.193125 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.193100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5cd229-4191-436f-bae3-841284e8f8f2-config-volume\") pod \"dns-default-ptvp7\" (UID: \"3b5cd229-4191-436f-bae3-841284e8f8f2\") " pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:19.193890 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.193428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.194116 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.194092    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9cb81b47-97bd-4561-85c3-054dbcf875b5-nginx-conf\") pod \"networking-console-plugin-79977f9844-m5m78\" (UID: \"9cb81b47-97bd-4561-85c3-054dbcf875b5\") " pod="openshift-network-console/networking-console-plugin-79977f9844-m5m78"
Sep 10 16:18:19.194964 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.194935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cb81b47-97bd-4561-85c3-054dbcf875b5-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-m5m78\" (UID: \"9cb81b47-97bd-4561-85c3-054dbcf875b5\") " pod="openshift-network-console/networking-console-plugin-79977f9844-m5m78"
Sep 10 16:18:19.195122 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.195099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b5cd229-4191-436f-bae3-841284e8f8f2-metrics-tls\") pod \"dns-default-ptvp7\" (UID: \"3b5cd229-4191-436f-bae3-841284e8f8f2\") " pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:19.195184 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.195153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0033469b-a67d-401e-8a8d-7603387b30d6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-865855d986-2dhqm\" (UID: \"0033469b-a67d-401e-8a8d-7603387b30d6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-865855d986-2dhqm"
Sep 10 16:18:19.195337 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.195318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8d04a4f4-b351-416a-9126-37c649ad0989-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.196129 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.196084    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/604b4c6c-a072-450a-95df-091c974db590-klusterlet-config\") pod \"klusterlet-addon-workmgr-76658f465f-jpdxk\" (UID: \"604b4c6c-a072-450a-95df-091c974db590\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"
Sep 10 16:18:19.196465 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.196437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8d04a4f4-b351-416a-9126-37c649ad0989-ca\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.196674 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.196657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.196716 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.196659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8d04a4f4-b351-416a-9126-37c649ad0989-hub\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.196858 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.196842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d04a4f4-b351-416a-9126-37c649ad0989-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.197128 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.197112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e5e1f877-d651-464b-9047-3c64e36a8f0d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-r2g6m\" (UID: \"e5e1f877-d651-464b-9047-3c64e36a8f0d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-r2g6m"
Sep 10 16:18:19.198976 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.198957    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2bbf6"
Sep 10 16:18:19.205107 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.205050    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdmmp\" (UniqueName: \"kubernetes.io/projected/0033469b-a67d-401e-8a8d-7603387b30d6-kube-api-access-vdmmp\") pod \"managed-serviceaccount-addon-agent-865855d986-2dhqm\" (UID: \"0033469b-a67d-401e-8a8d-7603387b30d6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-865855d986-2dhqm"
Sep 10 16:18:19.206824 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.206804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n28xb\" (UniqueName: \"kubernetes.io/projected/afe1b19f-f425-4126-87e5-5a5c0be3b4bd-kube-api-access-n28xb\") pod \"insights-runtime-extractor-k4nxn\" (UID: \"afe1b19f-f425-4126-87e5-5a5c0be3b4bd\") " pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.207061 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.207039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n785x\" (UniqueName: \"kubernetes.io/projected/3b5cd229-4191-436f-bae3-841284e8f8f2-kube-api-access-n785x\") pod \"dns-default-ptvp7\" (UID: \"3b5cd229-4191-436f-bae3-841284e8f8f2\") " pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:19.207245 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.207225    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7pvb\" (UniqueName: \"kubernetes.io/projected/604b4c6c-a072-450a-95df-091c974db590-kube-api-access-s7pvb\") pod \"klusterlet-addon-workmgr-76658f465f-jpdxk\" (UID: \"604b4c6c-a072-450a-95df-091c974db590\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"
Sep 10 16:18:19.207777 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.207754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2qz5\" (UniqueName: \"kubernetes.io/projected/8d04a4f4-b351-416a-9126-37c649ad0989-kube-api-access-c2qz5\") pod \"cluster-proxy-proxy-agent-597cfbc97-9s249\" (UID: \"8d04a4f4-b351-416a-9126-37c649ad0989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.293414 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.293388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mzgw\" (UniqueName: \"kubernetes.io/projected/ba733a9e-f54d-4aab-b83d-53d91ed0c306-kube-api-access-7mzgw\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.293522 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.293436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba733a9e-f54d-4aab-b83d-53d91ed0c306-bound-sa-token\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.293585 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.293513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba733a9e-f54d-4aab-b83d-53d91ed0c306-registry-tls\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.293622 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.293584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba733a9e-f54d-4aab-b83d-53d91ed0c306-image-registry-private-configuration\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.293622 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.293605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba733a9e-f54d-4aab-b83d-53d91ed0c306-installation-pull-secrets\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.293682 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.293624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba733a9e-f54d-4aab-b83d-53d91ed0c306-registry-certificates\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.293682 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.293675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba733a9e-f54d-4aab-b83d-53d91ed0c306-ca-trust-extracted\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.293745 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.293695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba733a9e-f54d-4aab-b83d-53d91ed0c306-trusted-ca\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.312461 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.312398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:19.322358 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.322278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-m5m78"
Sep 10 16:18:19.331227 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.331208    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-r2g6m"
Sep 10 16:18:19.348218 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.348192    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"
Sep 10 16:18:19.357740 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.357696    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k4nxn"
Sep 10 16:18:19.365483 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.365459    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-865855d986-2dhqm"
Sep 10 16:18:19.384830 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.384673    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"
Sep 10 16:18:19.395578 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.394353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba733a9e-f54d-4aab-b83d-53d91ed0c306-bound-sa-token\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.395578 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.394412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba733a9e-f54d-4aab-b83d-53d91ed0c306-registry-tls\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.395578 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.394453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba733a9e-f54d-4aab-b83d-53d91ed0c306-image-registry-private-configuration\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.395578 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.394483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba733a9e-f54d-4aab-b83d-53d91ed0c306-installation-pull-secrets\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.395578 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.394515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba733a9e-f54d-4aab-b83d-53d91ed0c306-registry-certificates\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.395578 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.394602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba733a9e-f54d-4aab-b83d-53d91ed0c306-ca-trust-extracted\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.395578 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.394629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba733a9e-f54d-4aab-b83d-53d91ed0c306-trusted-ca\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.395578 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.394700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mzgw\" (UniqueName: \"kubernetes.io/projected/ba733a9e-f54d-4aab-b83d-53d91ed0c306-kube-api-access-7mzgw\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.398249 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.396504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba733a9e-f54d-4aab-b83d-53d91ed0c306-ca-trust-extracted\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.398249 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.396899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba733a9e-f54d-4aab-b83d-53d91ed0c306-registry-certificates\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.398249 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.398158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba733a9e-f54d-4aab-b83d-53d91ed0c306-trusted-ca\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.402442 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.400338    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79d99c44cf-6kjvv"]
Sep 10 16:18:19.403978 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.403933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba733a9e-f54d-4aab-b83d-53d91ed0c306-registry-tls\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.414707 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.414514    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba733a9e-f54d-4aab-b83d-53d91ed0c306-image-registry-private-configuration\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.416130 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.416080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba733a9e-f54d-4aab-b83d-53d91ed0c306-bound-sa-token\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.417143 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.417032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba733a9e-f54d-4aab-b83d-53d91ed0c306-installation-pull-secrets\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.417860 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.417834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mzgw\" (UniqueName: \"kubernetes.io/projected/ba733a9e-f54d-4aab-b83d-53d91ed0c306-kube-api-access-7mzgw\") pod \"image-registry-5c6859976-hjrp4\" (UID: \"ba733a9e-f54d-4aab-b83d-53d91ed0c306\") " pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.424897 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.424714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:19.484854 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.483714    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2bbf6"]
Sep 10 16:18:19.586299 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.586231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2bbf6" event={"ID":"fc083413-78ac-40c4-b86c-16a0aec316f2","Type":"ContainerStarted","Data":"c9138acb1d46b8e8ede6cce07f33a3936ebaf9daabcca0404e92858bedc934bb"}
Sep 10 16:18:19.587759 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.587736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv" event={"ID":"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3","Type":"ContainerStarted","Data":"a94a82bb6c753b890e31f4ef5d85a129bc8a11629a6b7adb122e0ee78e1a46ba"}
Sep 10 16:18:19.587863 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.587773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv" event={"ID":"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3","Type":"ContainerStarted","Data":"3ecb82f6dbe202875817da5ad02cfa36e42a57bec2fe7f6c27bb8bdd11d252fb"}
Sep 10 16:18:19.587912 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.587898    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:19.598502 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.598463    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ptvp7"]
Sep 10 16:18:19.602844 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:18:19.602821    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5cd229_4191_436f_bae3_841284e8f8f2.slice/crio-777e20000026ea3f8b987238e72dafe6c2c4a0fefb90ba5e5b0eb9d76f85c363 WatchSource:0}: Error finding container 777e20000026ea3f8b987238e72dafe6c2c4a0fefb90ba5e5b0eb9d76f85c363: Status 404 returned error can't find the container with id 777e20000026ea3f8b987238e72dafe6c2c4a0fefb90ba5e5b0eb9d76f85c363
Sep 10 16:18:19.613219 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.613202    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-m5m78"]
Sep 10 16:18:19.615675 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:18:19.615644    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb81b47_97bd_4561_85c3_054dbcf875b5.slice/crio-6be09be05f79915c5e3dc647e33de85c82d4841b2cea2118b569cbc1110c46dc WatchSource:0}: Error finding container 6be09be05f79915c5e3dc647e33de85c82d4841b2cea2118b569cbc1110c46dc: Status 404 returned error can't find the container with id 6be09be05f79915c5e3dc647e33de85c82d4841b2cea2118b569cbc1110c46dc
Sep 10 16:18:19.616658 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.616474    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-r2g6m"]
Sep 10 16:18:19.618654 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.617951    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv" podStartSLOduration=44.617933468 podStartE2EDuration="44.617933468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:19.616015132 +0000 UTC m=+57.817057538" watchObservedRunningTime="2026-09-10 16:18:19.617933468 +0000 UTC m=+57.818975880"
Sep 10 16:18:19.662238 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.662209    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249"]
Sep 10 16:18:19.665239 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:18:19.665220    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d04a4f4_b351_416a_9126_37c649ad0989.slice/crio-91454352d54054aabeab6e01c99766f5b00632af739a28ecb71de650b7c76ba4 WatchSource:0}: Error finding container 91454352d54054aabeab6e01c99766f5b00632af739a28ecb71de650b7c76ba4: Status 404 returned error can't find the container with id 91454352d54054aabeab6e01c99766f5b00632af739a28ecb71de650b7c76ba4
Sep 10 16:18:19.696547 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.696504    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-865855d986-2dhqm"]
Sep 10 16:18:19.700015 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:18:19.699991    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0033469b_a67d_401e_8a8d_7603387b30d6.slice/crio-b1647c20be445bb48a1219ed336b224ce1bc03ceda7fcb4dc1d2788600bca010 WatchSource:0}: Error finding container b1647c20be445bb48a1219ed336b224ce1bc03ceda7fcb4dc1d2788600bca010: Status 404 returned error can't find the container with id b1647c20be445bb48a1219ed336b224ce1bc03ceda7fcb4dc1d2788600bca010
Sep 10 16:18:19.700368 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.700342    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k4nxn"]
Sep 10 16:18:19.705008 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:18:19.704715    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe1b19f_f425_4126_87e5_5a5c0be3b4bd.slice/crio-9f0ae960f16b06e6c517a1eca5668bb992abddcb1743dae7dd647526bd8490f4 WatchSource:0}: Error finding container 9f0ae960f16b06e6c517a1eca5668bb992abddcb1743dae7dd647526bd8490f4: Status 404 returned error can't find the container with id 9f0ae960f16b06e6c517a1eca5668bb992abddcb1743dae7dd647526bd8490f4
Sep 10 16:18:19.731793 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.731766    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"]
Sep 10 16:18:19.734609 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:19.734586    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c6859976-hjrp4"]
Sep 10 16:18:19.735166 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:18:19.735142    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604b4c6c_a072_450a_95df_091c974db590.slice/crio-e582bc51848ad380fa27fa6a015a059996230688bb5eb5bc531ff4d4f65b6d54 WatchSource:0}: Error finding container e582bc51848ad380fa27fa6a015a059996230688bb5eb5bc531ff4d4f65b6d54: Status 404 returned error can't find the container with id e582bc51848ad380fa27fa6a015a059996230688bb5eb5bc531ff4d4f65b6d54
Sep 10 16:18:19.737972 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:18:19.737950    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba733a9e_f54d_4aab_b83d_53d91ed0c306.slice/crio-948702be044b4d898895584514f9a6593b188aaf10d6e9d24247ae2bcaa02c95 WatchSource:0}: Error finding container 948702be044b4d898895584514f9a6593b188aaf10d6e9d24247ae2bcaa02c95: Status 404 returned error can't find the container with id 948702be044b4d898895584514f9a6593b188aaf10d6e9d24247ae2bcaa02c95
Sep 10 16:18:20.592705 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:20.592640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-r2g6m" event={"ID":"e5e1f877-d651-464b-9047-3c64e36a8f0d","Type":"ContainerStarted","Data":"5ed48f390ee793ea23250788be62025b0e9573628c1ccbe1d1e1bf906df2f691"}
Sep 10 16:18:20.594475 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:20.594414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ptvp7" event={"ID":"3b5cd229-4191-436f-bae3-841284e8f8f2","Type":"ContainerStarted","Data":"777e20000026ea3f8b987238e72dafe6c2c4a0fefb90ba5e5b0eb9d76f85c363"}
Sep 10 16:18:20.599989 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:20.599964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk" event={"ID":"604b4c6c-a072-450a-95df-091c974db590","Type":"ContainerStarted","Data":"e582bc51848ad380fa27fa6a015a059996230688bb5eb5bc531ff4d4f65b6d54"}
Sep 10 16:18:20.615518 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:20.615452    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-865855d986-2dhqm" event={"ID":"0033469b-a67d-401e-8a8d-7603387b30d6","Type":"ContainerStarted","Data":"b1647c20be445bb48a1219ed336b224ce1bc03ceda7fcb4dc1d2788600bca010"}
Sep 10 16:18:20.623707 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:20.623630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-m5m78" event={"ID":"9cb81b47-97bd-4561-85c3-054dbcf875b5","Type":"ContainerStarted","Data":"6be09be05f79915c5e3dc647e33de85c82d4841b2cea2118b569cbc1110c46dc"}
Sep 10 16:18:20.632774 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:20.632673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4nxn" event={"ID":"afe1b19f-f425-4126-87e5-5a5c0be3b4bd","Type":"ContainerStarted","Data":"0779781c2db889f04d67757d1d597cd8dde600fb175f4ca0221243f97e745651"}
Sep 10 16:18:20.632774 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:20.632727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4nxn" event={"ID":"afe1b19f-f425-4126-87e5-5a5c0be3b4bd","Type":"ContainerStarted","Data":"9f0ae960f16b06e6c517a1eca5668bb992abddcb1743dae7dd647526bd8490f4"}
Sep 10 16:18:20.634939 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:20.634884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249" event={"ID":"8d04a4f4-b351-416a-9126-37c649ad0989","Type":"ContainerStarted","Data":"91454352d54054aabeab6e01c99766f5b00632af739a28ecb71de650b7c76ba4"}
Sep 10 16:18:20.640350 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:20.640274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c6859976-hjrp4" event={"ID":"ba733a9e-f54d-4aab-b83d-53d91ed0c306","Type":"ContainerStarted","Data":"2d86cdfa1e1090c646dc16bee6c005a859dcb91d3455fa6a1e7fdcca7d5ad63a"}
Sep 10 16:18:20.640350 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:20.640314    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c6859976-hjrp4" event={"ID":"ba733a9e-f54d-4aab-b83d-53d91ed0c306","Type":"ContainerStarted","Data":"948702be044b4d898895584514f9a6593b188aaf10d6e9d24247ae2bcaa02c95"}
Sep 10 16:18:20.640607 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:20.640559    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:20.663117 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:20.663050    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c6859976-hjrp4" podStartSLOduration=1.663032212 podStartE2EDuration="1.663032212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:20.66086034 +0000 UTC m=+58.861902756" watchObservedRunningTime="2026-09-10 16:18:20.663032212 +0000 UTC m=+58.864074625"
Sep 10 16:18:21.655919 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:21.655812    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4nxn" event={"ID":"afe1b19f-f425-4126-87e5-5a5c0be3b4bd","Type":"ContainerStarted","Data":"5462d0cbeaaf6105558f71c38b811c7447f78ac84437bb47526407e648f47302"}
Sep 10 16:18:23.526748 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:23.526715    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbjsw"
Sep 10 16:18:28.080505 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.080466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs\") pod \"network-metrics-daemon-lnx4z\" (UID: \"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10\") " pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:18:28.080993 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.080550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:18:28.083232 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.083201    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 16:18:28.083730 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.083710    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 16:18:28.093836 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.093812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2115e2-15e0-4fe4-9bfd-1158f59a0c10-metrics-certs\") pod \"network-metrics-daemon-lnx4z\" (UID: \"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10\") " pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:18:28.093925 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.093863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25606248-47cf-4de0-aab7-48d1123e9901-original-pull-secret\") pod \"global-pull-secret-syncer-dc7jq\" (UID: \"25606248-47cf-4de0-aab7-48d1123e9901\") " pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:18:28.160439 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.160409    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j8r7d\""
Sep 10 16:18:28.162158 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.162138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dc7jq"
Sep 10 16:18:28.168678 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.168656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnx4z"
Sep 10 16:18:28.181590 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.181560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpwq\" (UniqueName: \"kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq\") pod \"network-check-target-vzlvs\" (UID: \"218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f\") " pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:18:28.183755 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.183732    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 16:18:28.194350 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.194327    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 16:18:28.205355 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.205329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvpwq\" (UniqueName: \"kubernetes.io/projected/218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f-kube-api-access-nvpwq\") pod \"network-check-target-vzlvs\" (UID: \"218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f\") " pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:18:28.261604 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.261582    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-htvxx\""
Sep 10 16:18:28.269660 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.269640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:18:28.963039 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.963012    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lnx4z"]
Sep 10 16:18:28.968740 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.967635    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vzlvs"]
Sep 10 16:18:28.968740 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:18:28.968284    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2115e2_15e0_4fe4_9bfd_1158f59a0c10.slice/crio-4a86774756e0d616e9ac993c476504302483a3224e7ee62a541865c3ba8bc3ea WatchSource:0}: Error finding container 4a86774756e0d616e9ac993c476504302483a3224e7ee62a541865c3ba8bc3ea: Status 404 returned error can't find the container with id 4a86774756e0d616e9ac993c476504302483a3224e7ee62a541865c3ba8bc3ea
Sep 10 16:18:28.968740 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:28.968354    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dc7jq"]
Sep 10 16:18:28.970887 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:18:28.970867    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25606248_47cf_4de0_aab7_48d1123e9901.slice/crio-9189c50ef6a2de43140f7b3b8a2c4227cdedc5958c8ff6e19cc674847f7ada41 WatchSource:0}: Error finding container 9189c50ef6a2de43140f7b3b8a2c4227cdedc5958c8ff6e19cc674847f7ada41: Status 404 returned error can't find the container with id 9189c50ef6a2de43140f7b3b8a2c4227cdedc5958c8ff6e19cc674847f7ada41
Sep 10 16:18:28.984602 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:18:28.984581    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218c15b9_ca62_4d71_b0e9_f7c7cb65ee3f.slice/crio-47a58b42febe4809ff2a1c775c126df060cadf678c7e0515b07f442a186aaa6c WatchSource:0}: Error finding container 47a58b42febe4809ff2a1c775c126df060cadf678c7e0515b07f442a186aaa6c: Status 404 returned error can't find the container with id 47a58b42febe4809ff2a1c775c126df060cadf678c7e0515b07f442a186aaa6c
Sep 10 16:18:29.687349 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.687313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-r2g6m" event={"ID":"e5e1f877-d651-464b-9047-3c64e36a8f0d","Type":"ContainerStarted","Data":"a4af695562dedcd36073a9bb28fb066841147f4b3acff8403c0d62bb3a5fd430"}
Sep 10 16:18:29.687910 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.687704    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-r2g6m"
Sep 10 16:18:29.691227 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.690724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ptvp7" event={"ID":"3b5cd229-4191-436f-bae3-841284e8f8f2","Type":"ContainerStarted","Data":"b8fe555b90d230ce761dbec6f8915d50cd9d743fd9f3597fff6165f352f374d2"}
Sep 10 16:18:29.691227 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.690775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ptvp7" event={"ID":"3b5cd229-4191-436f-bae3-841284e8f8f2","Type":"ContainerStarted","Data":"5e084d4efe339d7fc7153a3ebf7f9a5e1e8599d0b6472337897f01d8c6f4fddb"}
Sep 10 16:18:29.691227 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.691167    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:29.692686 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.692658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dc7jq" event={"ID":"25606248-47cf-4de0-aab7-48d1123e9901","Type":"ContainerStarted","Data":"9189c50ef6a2de43140f7b3b8a2c4227cdedc5958c8ff6e19cc674847f7ada41"}
Sep 10 16:18:29.694292 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.694269    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-r2g6m"
Sep 10 16:18:29.694786 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.694757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnx4z" event={"ID":"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10","Type":"ContainerStarted","Data":"4a86774756e0d616e9ac993c476504302483a3224e7ee62a541865c3ba8bc3ea"}
Sep 10 16:18:29.696854 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.696833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk" event={"ID":"604b4c6c-a072-450a-95df-091c974db590","Type":"ContainerStarted","Data":"a0124b794d676059366f842f686c7ca2bdeeb3cc578ec1252bc2b61de42c5c47"}
Sep 10 16:18:29.696958 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.696877    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"
Sep 10 16:18:29.698622 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.698563    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk"
Sep 10 16:18:29.700347 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.700304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-865855d986-2dhqm" event={"ID":"0033469b-a67d-401e-8a8d-7603387b30d6","Type":"ContainerStarted","Data":"2d1c533c49c99ef1321d755018b474302ffc231a6ebe3b388a3db63b2c885e0f"}
Sep 10 16:18:29.703617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.703216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-m5m78" event={"ID":"9cb81b47-97bd-4561-85c3-054dbcf875b5","Type":"ContainerStarted","Data":"41c37267a1011bc0ca0a83ff112b4e9ebded719575de721a152149576f03d2c3"}
Sep 10 16:18:29.705134 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.705109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vzlvs" event={"ID":"218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f","Type":"ContainerStarted","Data":"47a58b42febe4809ff2a1c775c126df060cadf678c7e0515b07f442a186aaa6c"}
Sep 10 16:18:29.707763 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.706576    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-r2g6m" podStartSLOduration=2.674306288 podStartE2EDuration="11.706560273s" totalImagesPullingTime="9.032253985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:29.706043922 +0000 UTC m=+67.907086340" watchObservedRunningTime="2026-09-10 16:18:29.706560273 +0000 UTC m=+67.907602684"
Sep 10 16:18:29.710042 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.710016    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2bbf6" event={"ID":"fc083413-78ac-40c4-b86c-16a0aec316f2","Type":"ContainerStarted","Data":"bfb5ce620451b5dc4f62c86a50787e1fb2e29834581d45b375c784425a5cf755"}
Sep 10 16:18:29.712810 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.712781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4nxn" event={"ID":"afe1b19f-f425-4126-87e5-5a5c0be3b4bd","Type":"ContainerStarted","Data":"0d94d7a3c43a594139f40f1eeb328639c3ee62bcf0d9685eff9f3488e77c7b96"}
Sep 10 16:18:29.715569 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.715547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249" event={"ID":"8d04a4f4-b351-416a-9126-37c649ad0989","Type":"ContainerStarted","Data":"a7a073e8d4af59e9091978dff90707c8a9ca74176fca40c97bec3ad6cec25550"}
Sep 10 16:18:29.715669 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.715584    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249" event={"ID":"8d04a4f4-b351-416a-9126-37c649ad0989","Type":"ContainerStarted","Data":"5ecb6b471f8fcbf54dc1c60339ece27ddba9c234e18ca18ab2a50eda300e6bf6"}
Sep 10 16:18:29.715669 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.715650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249" event={"ID":"8d04a4f4-b351-416a-9126-37c649ad0989","Type":"ContainerStarted","Data":"b8cb3a2be2c6ea03fe9f695eccfb122cbed8af74ab8045853f90b42cea31de8f"}
Sep 10 16:18:29.740361 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.739640    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-m5m78" podStartSLOduration=2.70417673 podStartE2EDuration="11.739621285s" totalImagesPullingTime="9.035444555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:29.723739168 +0000 UTC m=+67.924781580" watchObservedRunningTime="2026-09-10 16:18:29.739621285 +0000 UTC m=+67.940663697"
Sep 10 16:18:29.760228 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.759874    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-865855d986-2dhqm" podStartSLOduration=2.782461659 podStartE2EDuration="11.759856249s" totalImagesPullingTime="8.97739459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:29.75846216 +0000 UTC m=+67.959504574" watchObservedRunningTime="2026-09-10 16:18:29.759856249 +0000 UTC m=+67.960898656"
Sep 10 16:18:29.779220 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.779176    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76658f465f-jpdxk" podStartSLOduration=2.825412913 podStartE2EDuration="11.779164057s" totalImagesPullingTime="8.953751144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:29.778110081 +0000 UTC m=+67.979152494" watchObservedRunningTime="2026-09-10 16:18:29.779164057 +0000 UTC m=+67.980206485"
Sep 10 16:18:29.801474 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.801429    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ptvp7" podStartSLOduration=2.75019046 podStartE2EDuration="11.801410682s" totalImagesPullingTime="9.051220222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:29.799954658 +0000 UTC m=+68.000997082" watchObservedRunningTime="2026-09-10 16:18:29.801410682 +0000 UTC m=+68.002453094"
Sep 10 16:18:29.821488 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.820542    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k4nxn" podStartSLOduration=3.062720095 podStartE2EDuration="11.820511944s" totalImagesPullingTime="8.757791849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:18:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:29.819803217 +0000 UTC m=+68.020845665" watchObservedRunningTime="2026-09-10 16:18:29.820511944 +0000 UTC m=+68.021554358"
Sep 10 16:18:29.843617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.843247    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-597cfbc97-9s249" podStartSLOduration=2.820410199 podStartE2EDuration="11.843230073s" totalImagesPullingTime="9.022819874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:29.842250581 +0000 UTC m=+68.043292993" watchObservedRunningTime="2026-09-10 16:18:29.843230073 +0000 UTC m=+68.044272485"
Sep 10 16:18:29.859873 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:29.859826    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2bbf6" podStartSLOduration=2.6759180689999997 podStartE2EDuration="11.859814145s" totalImagesPullingTime="9.183896076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:29.858993342 +0000 UTC m=+68.060035747" watchObservedRunningTime="2026-09-10 16:18:29.859814145 +0000 UTC m=+68.060856559"
Sep 10 16:18:30.722211 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:30.722166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnx4z" event={"ID":"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10","Type":"ContainerStarted","Data":"be4e8b5db5ddb31287958e0299d845c87d238dadda4d5db5b699f8d0171d0ced"}
Sep 10 16:18:30.722681 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:30.722234    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnx4z" event={"ID":"0b2115e2-15e0-4fe4-9bfd-1158f59a0c10","Type":"ContainerStarted","Data":"075eac2ba5d4ed3c71ac1b3696e1933954e1c9f75031b028e5b3a6a75265014c"}
Sep 10 16:18:30.740205 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:30.740063    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lnx4z" podStartSLOduration=66.545811122 podStartE2EDuration="1m7.740041347s" totalImagesPullingTime="1.194230225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:30.738691612 +0000 UTC m=+68.939734023" watchObservedRunningTime="2026-09-10 16:18:30.740041347 +0000 UTC m=+68.941083759"
Sep 10 16:18:34.553426 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.553392    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c4jjk"]
Sep 10 16:18:34.574719 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.574693    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.576948 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.576923    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 16:18:34.577058 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.577004    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 16:18:34.577684 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.577667    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 16:18:34.577784 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.577727    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 16:18:34.577853 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.577828    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 16:18:34.577913 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.577874    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gmj9w\""
Sep 10 16:18:34.577966 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.577915    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 16:18:34.637920 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.637894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-wtmp\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.638020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.637924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc67c\" (UniqueName: \"kubernetes.io/projected/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-kube-api-access-qc67c\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.638020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.637945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.638020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.637971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-textfile\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.638020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.637993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-sys\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.638173 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.638027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-tls\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.638173 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.638058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-metrics-client-ca\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.638173 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.638093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-root\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.638173 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.638115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.738454 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.738432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-metrics-client-ca\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.738654 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.738475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-root\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.738654 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.738544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-root\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.738654 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.738565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.738654 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.738611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-wtmp\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.738654 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.738627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc67c\" (UniqueName: \"kubernetes.io/projected/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-kube-api-access-qc67c\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.738911 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.738659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.738911 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.738682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-textfile\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.738911 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.738698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-sys\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.738911 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.738719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-tls\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.738911 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.738792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-wtmp\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.739144 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.739107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-metrics-client-ca\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.739256 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.739193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-textfile\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.739256 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.739205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.739256 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.739200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-sys\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.741314 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.741286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-tls\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.741469 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.741446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.743575 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.743521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dc7jq" event={"ID":"25606248-47cf-4de0-aab7-48d1123e9901","Type":"ContainerStarted","Data":"55c1145b6ed91f87ab18a9da49613590945a540a05dcfeec9d4d121533e9119a"}
Sep 10 16:18:34.745324 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.745299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vzlvs" event={"ID":"218c15b9-ca62-4d71-b0e9-f7c7cb65ee3f","Type":"ContainerStarted","Data":"26cb34c5353d06e6937175c5e330287dc5da19df1b06a865b2b093d5af7218c7"}
Sep 10 16:18:34.745412 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.745372    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:18:34.747779 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.747755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc67c\" (UniqueName: \"kubernetes.io/projected/8b1b78b7-342c-4e54-bc3c-2771f3ff25cc-kube-api-access-qc67c\") pod \"node-exporter-c4jjk\" (UID: \"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc\") " pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.763430 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.763389    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dc7jq" podStartSLOduration=66.763637269 podStartE2EDuration="1m11.763376443s" totalImagesPullingTime="4.999739174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:34.762767584 +0000 UTC m=+72.963809995" watchObservedRunningTime="2026-09-10 16:18:34.763376443 +0000 UTC m=+72.964418853"
Sep 10 16:18:34.779341 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.779308    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vzlvs" podStartSLOduration=66.803308003 podStartE2EDuration="1m11.779295993s" totalImagesPullingTime="4.97598799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:34.778595852 +0000 UTC m=+72.979638265" watchObservedRunningTime="2026-09-10 16:18:34.779295993 +0000 UTC m=+72.980338403"
Sep 10 16:18:34.883470 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:34.883425    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c4jjk"
Sep 10 16:18:34.915692 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:18:34.915670    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1b78b7_342c_4e54_bc3c_2771f3ff25cc.slice/crio-e05b9293da1728699e502e823d6f73ebe43646beb4d623e6e7145679cb60a549 WatchSource:0}: Error finding container e05b9293da1728699e502e823d6f73ebe43646beb4d623e6e7145679cb60a549: Status 404 returned error can't find the container with id e05b9293da1728699e502e823d6f73ebe43646beb4d623e6e7145679cb60a549
Sep 10 16:18:35.750149 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:35.750116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4jjk" event={"ID":"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc","Type":"ContainerStarted","Data":"e05b9293da1728699e502e823d6f73ebe43646beb4d623e6e7145679cb60a549"}
Sep 10 16:18:36.755212 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:36.755175    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b1b78b7-342c-4e54-bc3c-2771f3ff25cc" containerID="b3a78bc8d4892fa26482a120744b7fa60e5210c665c5487763598fa7dd321cfa" exitCode=0
Sep 10 16:18:36.755693 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:36.755259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4jjk" event={"ID":"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc","Type":"ContainerDied","Data":"b3a78bc8d4892fa26482a120744b7fa60e5210c665c5487763598fa7dd321cfa"}
Sep 10 16:18:37.761454 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:37.761419    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4jjk" event={"ID":"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc","Type":"ContainerStarted","Data":"9f0130c5103cd22ea90900596ff2caa89307947a993c019fc2b1e13c94f5ef8d"}
Sep 10 16:18:37.761839 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:37.761500    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4jjk" event={"ID":"8b1b78b7-342c-4e54-bc3c-2771f3ff25cc","Type":"ContainerStarted","Data":"3337b4784be22db36fc748ab4b77c97c33154ac1b5161f7337ddda2fa8bfed9a"}
Sep 10 16:18:37.788839 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:37.788784    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c4jjk" podStartSLOduration=2.676006216 podStartE2EDuration="3.788767156s" totalImagesPullingTime="1.052807296s" totalInitContainerRuntime="59.953644ms" isStatefulPod=true podCreationTimestamp="2026-09-10 16:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:37.787977924 +0000 UTC m=+75.989020357" watchObservedRunningTime="2026-09-10 16:18:37.788767156 +0000 UTC m=+75.989809569"
Sep 10 16:18:39.593578 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:39.593550    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:39.725257 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:39.725227    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ptvp7"
Sep 10 16:18:41.659816 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:41.659786    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c6859976-hjrp4"
Sep 10 16:18:44.608838 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.608719    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv" podUID="26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3" containerName="registry" containerID="cri-o://a94a82bb6c753b890e31f4ef5d85a129bc8a11629a6b7adb122e0ee78e1a46ba" gracePeriod=30
Sep 10 16:18:44.785913 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.785878    2580 generic.go:356] "Generic (PLEG): container finished" podID="26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3" containerID="a94a82bb6c753b890e31f4ef5d85a129bc8a11629a6b7adb122e0ee78e1a46ba" exitCode=0
Sep 10 16:18:44.786030 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.785955    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv" event={"ID":"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3","Type":"ContainerDied","Data":"a94a82bb6c753b890e31f4ef5d85a129bc8a11629a6b7adb122e0ee78e1a46ba"}
Sep 10 16:18:44.843383 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.843360    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:44.911578 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.911485    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-registry-certificates\") pod \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") "
Sep 10 16:18:44.911578 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.911549    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-trusted-ca\") pod \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") "
Sep 10 16:18:44.911794 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.911605    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-installation-pull-secrets\") pod \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") "
Sep 10 16:18:44.911794 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.911628    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-registry-tls\") pod \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") "
Sep 10 16:18:44.911794 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.911673    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-image-registry-private-configuration\") pod \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") "
Sep 10 16:18:44.911794 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.911705    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-bound-sa-token\") pod \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") "
Sep 10 16:18:44.911794 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.911726    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qccx7\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-kube-api-access-qccx7\") pod \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") "
Sep 10 16:18:44.912028 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.911828    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-ca-trust-extracted\") pod \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\" (UID: \"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3\") "
Sep 10 16:18:44.912028 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.911957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3" (UID: "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:18:44.912115 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.912023    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3" (UID: "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:18:44.912115 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.912046    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-registry-certificates\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:18:44.914241 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.914199    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-kube-api-access-qccx7" (OuterVolumeSpecName: "kube-api-access-qccx7") pod "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3" (UID: "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3"). InnerVolumeSpecName "kube-api-access-qccx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:18:44.914364 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.914256    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3" (UID: "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:18:44.914513 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.914484    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3" (UID: "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:18:44.914640 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.914515    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3" (UID: "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:18:44.914640 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.914574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3" (UID: "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:18:44.920767 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:44.920742    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3" (UID: "26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:18:45.012919 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:45.012876    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-ca-trust-extracted\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:18:45.012919 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:45.012912    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-trusted-ca\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:18:45.012919 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:45.012922    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-installation-pull-secrets\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:18:45.013149 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:45.012931    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-registry-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:18:45.013149 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:45.012954    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-image-registry-private-configuration\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:18:45.013149 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:45.012963    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-bound-sa-token\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:18:45.013149 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:45.012972    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qccx7\" (UniqueName: \"kubernetes.io/projected/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3-kube-api-access-qccx7\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:18:45.790876 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:45.790842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv" event={"ID":"26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3","Type":"ContainerDied","Data":"3ecb82f6dbe202875817da5ad02cfa36e42a57bec2fe7f6c27bb8bdd11d252fb"}
Sep 10 16:18:45.790876 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:45.790875    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d99c44cf-6kjvv"
Sep 10 16:18:45.791310 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:45.790901    2580 scope.go:117] "RemoveContainer" containerID="a94a82bb6c753b890e31f4ef5d85a129bc8a11629a6b7adb122e0ee78e1a46ba"
Sep 10 16:18:45.811317 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:45.811294    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79d99c44cf-6kjvv"]
Sep 10 16:18:45.817107 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:45.817079    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79d99c44cf-6kjvv"]
Sep 10 16:18:46.253623 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:18:46.253514    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3" path="/var/lib/kubelet/pods/26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3/volumes"
Sep 10 16:19:05.753111 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:19:05.753078    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vzlvs"
Sep 10 16:22:14.812630 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.812591    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"]
Sep 10 16:22:14.813091 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.812895    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3" containerName="registry"
Sep 10 16:22:14.813091 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.812908    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3" containerName="registry"
Sep 10 16:22:14.813091 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.812959    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="26451ef2-d2cf-4cd0-a94c-b36fb9d4f0c3" containerName="registry"
Sep 10 16:22:14.814734 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.814719    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:14.819901 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.819873    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 16:22:14.820044 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.819914    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 16:22:14.820044 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.819930    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 16:22:14.820044 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.820039    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tbzcd\""
Sep 10 16:22:14.820301 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.820280    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 16:22:14.841300 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.841277    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"]
Sep 10 16:22:14.969597 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.969563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81302c0c-3dae-4657-9a6d-af3540e0e8ac-manifests\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:14.969767 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.969603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81302c0c-3dae-4657-9a6d-af3540e0e8ac-tmp\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:14.969767 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.969632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81302c0c-3dae-4657-9a6d-af3540e0e8ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:14.969767 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.969675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qh2\" (UniqueName: \"kubernetes.io/projected/81302c0c-3dae-4657-9a6d-af3540e0e8ac-kube-api-access-d5qh2\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:14.969767 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:14.969748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81302c0c-3dae-4657-9a6d-af3540e0e8ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:15.070960 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:15.070894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81302c0c-3dae-4657-9a6d-af3540e0e8ac-manifests\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:15.070960 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:15.070928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81302c0c-3dae-4657-9a6d-af3540e0e8ac-tmp\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:15.070960 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:15.070946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81302c0c-3dae-4657-9a6d-af3540e0e8ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:15.071168 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:15.070980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qh2\" (UniqueName: \"kubernetes.io/projected/81302c0c-3dae-4657-9a6d-af3540e0e8ac-kube-api-access-d5qh2\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:15.071168 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:15.071005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81302c0c-3dae-4657-9a6d-af3540e0e8ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:15.071353 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:15.071317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81302c0c-3dae-4657-9a6d-af3540e0e8ac-manifests\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:15.073264 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:15.073242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81302c0c-3dae-4657-9a6d-af3540e0e8ac-tmp\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:15.073610 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:15.073590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81302c0c-3dae-4657-9a6d-af3540e0e8ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:15.073661 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:15.073590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81302c0c-3dae-4657-9a6d-af3540e0e8ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:15.080619 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:15.080601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qh2\" (UniqueName: \"kubernetes.io/projected/81302c0c-3dae-4657-9a6d-af3540e0e8ac-kube-api-access-d5qh2\") pod \"opendatahub-operator-controller-manager-57b77968d9-5ht8c\" (UID: \"81302c0c-3dae-4657-9a6d-af3540e0e8ac\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:15.124605 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:15.124570    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:15.405304 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:15.405228    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"]
Sep 10 16:22:15.408454 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:22:15.408427    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81302c0c_3dae_4657_9a6d_af3540e0e8ac.slice/crio-496a9319a369f83f96da38d5b41cf9d0422f57201cbb2b0b569916b91f3e8699 WatchSource:0}: Error finding container 496a9319a369f83f96da38d5b41cf9d0422f57201cbb2b0b569916b91f3e8699: Status 404 returned error can't find the container with id 496a9319a369f83f96da38d5b41cf9d0422f57201cbb2b0b569916b91f3e8699
Sep 10 16:22:15.475586 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:15.475554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c" event={"ID":"81302c0c-3dae-4657-9a6d-af3540e0e8ac","Type":"ContainerStarted","Data":"496a9319a369f83f96da38d5b41cf9d0422f57201cbb2b0b569916b91f3e8699"}
Sep 10 16:22:18.490580 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:18.490547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c" event={"ID":"81302c0c-3dae-4657-9a6d-af3540e0e8ac","Type":"ContainerStarted","Data":"f276ec6438b6663c5e7a50038363ee3503e064f7ba645913c3f5397557618183"}
Sep 10 16:22:20.498584 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:20.498523    2580 generic.go:356] "Generic (PLEG): container finished" podID="81302c0c-3dae-4657-9a6d-af3540e0e8ac" containerID="f276ec6438b6663c5e7a50038363ee3503e064f7ba645913c3f5397557618183" exitCode=0
Sep 10 16:22:20.498965 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:20.498597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c" event={"ID":"81302c0c-3dae-4657-9a6d-af3540e0e8ac","Type":"ContainerDied","Data":"f276ec6438b6663c5e7a50038363ee3503e064f7ba645913c3f5397557618183"}
Sep 10 16:22:21.504291 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:21.504208    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c" event={"ID":"81302c0c-3dae-4657-9a6d-af3540e0e8ac","Type":"ContainerStarted","Data":"71dd1ab0f22e95a1932fd54014cf7df29b3e7695a8ab875cddbdf78c1f248563"}
Sep 10 16:22:21.504665 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:21.504439    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:21.533939 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:21.533890    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c" podStartSLOduration=2.84572912 podStartE2EDuration="7.533875809s" totalImagesPullingTime="3.676945092s" totalInitContainerRuntime="1.011201597s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:22:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:22:21.533171452 +0000 UTC m=+299.734213877" watchObservedRunningTime="2026-09-10 16:22:21.533875809 +0000 UTC m=+299.734918202"
Sep 10 16:22:22.209127 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:22.209100    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 16:22:32.509571 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:32.509526    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-5ht8c"
Sep 10 16:22:40.138668 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.138637    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"]
Sep 10 16:22:40.140633 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.140617    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"
Sep 10 16:22:40.143037 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.143012    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 16:22:40.143153 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.143052    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 16:22:40.143153 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.143088    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 16:22:40.143328 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.143314    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 16:22:40.151523 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.151502    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"]
Sep 10 16:22:40.248816 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.248782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-tls-certs\") pod \"kube-auth-proxy-5cfcc4b4c4-xdjsr\" (UID: \"d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f\") " pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"
Sep 10 16:22:40.248971 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.248823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-tmp\") pod \"kube-auth-proxy-5cfcc4b4c4-xdjsr\" (UID: \"d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f\") " pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"
Sep 10 16:22:40.248971 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.248863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz959\" (UniqueName: \"kubernetes.io/projected/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-kube-api-access-pz959\") pod \"kube-auth-proxy-5cfcc4b4c4-xdjsr\" (UID: \"d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f\") " pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"
Sep 10 16:22:40.350111 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.350086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-tls-certs\") pod \"kube-auth-proxy-5cfcc4b4c4-xdjsr\" (UID: \"d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f\") " pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"
Sep 10 16:22:40.350249 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.350120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-tmp\") pod \"kube-auth-proxy-5cfcc4b4c4-xdjsr\" (UID: \"d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f\") " pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"
Sep 10 16:22:40.350249 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.350159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz959\" (UniqueName: \"kubernetes.io/projected/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-kube-api-access-pz959\") pod \"kube-auth-proxy-5cfcc4b4c4-xdjsr\" (UID: \"d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f\") " pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"
Sep 10 16:22:40.350249 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:22:40.350232    2580 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 16:22:40.350355 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:22:40.350324    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-tls-certs podName:d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f nodeName:}" failed. No retries permitted until 2026-09-10 16:22:40.850304768 +0000 UTC m=+319.051347160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-tls-certs") pod "kube-auth-proxy-5cfcc4b4c4-xdjsr" (UID: "d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f") : secret "kube-auth-proxy-tls" not found
Sep 10 16:22:40.352353 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.352330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-tmp\") pod \"kube-auth-proxy-5cfcc4b4c4-xdjsr\" (UID: \"d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f\") " pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"
Sep 10 16:22:40.359382 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.359354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz959\" (UniqueName: \"kubernetes.io/projected/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-kube-api-access-pz959\") pod \"kube-auth-proxy-5cfcc4b4c4-xdjsr\" (UID: \"d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f\") " pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"
Sep 10 16:22:40.855054 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:40.855011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-tls-certs\") pod \"kube-auth-proxy-5cfcc4b4c4-xdjsr\" (UID: \"d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f\") " pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"
Sep 10 16:22:40.855238 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:22:40.855163    2580 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 16:22:40.855238 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:22:40.855227    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-tls-certs podName:d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f nodeName:}" failed. No retries permitted until 2026-09-10 16:22:41.855210262 +0000 UTC m=+320.056252651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-tls-certs") pod "kube-auth-proxy-5cfcc4b4c4-xdjsr" (UID: "d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f") : secret "kube-auth-proxy-tls" not found
Sep 10 16:22:41.862836 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:41.862802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-tls-certs\") pod \"kube-auth-proxy-5cfcc4b4c4-xdjsr\" (UID: \"d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f\") " pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"
Sep 10 16:22:41.865201 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:41.865177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f-tls-certs\") pod \"kube-auth-proxy-5cfcc4b4c4-xdjsr\" (UID: \"d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f\") " pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"
Sep 10 16:22:41.949566 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:41.949527    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"
Sep 10 16:22:42.229248 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:42.229070    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr"]
Sep 10 16:22:42.231715 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:22:42.231686    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d2caa6_f481_4ea7_99c9_b1aa91ddc42f.slice/crio-1b785684582064eabe714ba32a6e814e0885400848604fa044d5507c0a7d6075 WatchSource:0}: Error finding container 1b785684582064eabe714ba32a6e814e0885400848604fa044d5507c0a7d6075: Status 404 returned error can't find the container with id 1b785684582064eabe714ba32a6e814e0885400848604fa044d5507c0a7d6075
Sep 10 16:22:42.233505 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:42.233488    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 16:22:42.578482 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:42.578445    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr" event={"ID":"d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f","Type":"ContainerStarted","Data":"1b785684582064eabe714ba32a6e814e0885400848604fa044d5507c0a7d6075"}
Sep 10 16:22:45.431026 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:45.430991    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n9ffr"]
Sep 10 16:22:45.433163 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:45.433146    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-n9ffr"
Sep 10 16:22:45.435606 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:45.435588    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 10 16:22:45.435678 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:45.435628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-rbp4c\""
Sep 10 16:22:45.445463 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:45.445438    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n9ffr"]
Sep 10 16:22:45.490294 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:45.490263    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb4rf\" (UniqueName: \"kubernetes.io/projected/14f7f3f3-6711-4eee-9ef1-34a8100a357f-kube-api-access-cb4rf\") pod \"odh-model-controller-7998c67dc4-n9ffr\" (UID: \"14f7f3f3-6711-4eee-9ef1-34a8100a357f\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9ffr"
Sep 10 16:22:45.490396 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:45.490308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14f7f3f3-6711-4eee-9ef1-34a8100a357f-cert\") pod \"odh-model-controller-7998c67dc4-n9ffr\" (UID: \"14f7f3f3-6711-4eee-9ef1-34a8100a357f\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9ffr"
Sep 10 16:22:45.591563 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:45.591514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb4rf\" (UniqueName: \"kubernetes.io/projected/14f7f3f3-6711-4eee-9ef1-34a8100a357f-kube-api-access-cb4rf\") pod \"odh-model-controller-7998c67dc4-n9ffr\" (UID: \"14f7f3f3-6711-4eee-9ef1-34a8100a357f\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9ffr"
Sep 10 16:22:45.591723 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:45.591585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14f7f3f3-6711-4eee-9ef1-34a8100a357f-cert\") pod \"odh-model-controller-7998c67dc4-n9ffr\" (UID: \"14f7f3f3-6711-4eee-9ef1-34a8100a357f\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9ffr"
Sep 10 16:22:45.591723 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:22:45.591705    2580 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 10 16:22:45.591844 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:22:45.591780    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14f7f3f3-6711-4eee-9ef1-34a8100a357f-cert podName:14f7f3f3-6711-4eee-9ef1-34a8100a357f nodeName:}" failed. No retries permitted until 2026-09-10 16:22:46.09175695 +0000 UTC m=+324.292799361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/14f7f3f3-6711-4eee-9ef1-34a8100a357f-cert") pod "odh-model-controller-7998c67dc4-n9ffr" (UID: "14f7f3f3-6711-4eee-9ef1-34a8100a357f") : secret "odh-model-controller-webhook-cert" not found
Sep 10 16:22:45.595076 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:45.595044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr" event={"ID":"d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f","Type":"ContainerStarted","Data":"04f091c19bfea61c0fc57cdc8629affb2858ea7f98fdc93ac2ec5cdd499ca1d4"}
Sep 10 16:22:45.603177 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:45.603153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb4rf\" (UniqueName: \"kubernetes.io/projected/14f7f3f3-6711-4eee-9ef1-34a8100a357f-kube-api-access-cb4rf\") pod \"odh-model-controller-7998c67dc4-n9ffr\" (UID: \"14f7f3f3-6711-4eee-9ef1-34a8100a357f\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9ffr"
Sep 10 16:22:45.615798 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:45.615754    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cfcc4b4c4-xdjsr" podStartSLOduration=3.024018086 podStartE2EDuration="5.615740584s" totalImagesPullingTime="2.591722498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:22:45.614039035 +0000 UTC m=+323.815081458" watchObservedRunningTime="2026-09-10 16:22:45.615740584 +0000 UTC m=+323.816783000"
Sep 10 16:22:46.095309 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.095278    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14f7f3f3-6711-4eee-9ef1-34a8100a357f-cert\") pod \"odh-model-controller-7998c67dc4-n9ffr\" (UID: \"14f7f3f3-6711-4eee-9ef1-34a8100a357f\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9ffr"
Sep 10 16:22:46.097844 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.097815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14f7f3f3-6711-4eee-9ef1-34a8100a357f-cert\") pod \"odh-model-controller-7998c67dc4-n9ffr\" (UID: \"14f7f3f3-6711-4eee-9ef1-34a8100a357f\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9ffr"
Sep 10 16:22:46.197867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.197834    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7rtlr"]
Sep 10 16:22:46.200084 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.200065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7rtlr"
Sep 10 16:22:46.204398 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.204377    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-4pnjt\""
Sep 10 16:22:46.204625 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.204610    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 16:22:46.215144 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.215118    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7rtlr"]
Sep 10 16:22:46.297031 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.297002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e4d234-8c3e-4ad2-8931-99af2824b387-tls-certs\") pod \"model-serving-api-54d89bd5fc-7rtlr\" (UID: \"11e4d234-8c3e-4ad2-8931-99af2824b387\") " pod="opendatahub/model-serving-api-54d89bd5fc-7rtlr"
Sep 10 16:22:46.297167 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.297044    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw8mt\" (UniqueName: \"kubernetes.io/projected/11e4d234-8c3e-4ad2-8931-99af2824b387-kube-api-access-sw8mt\") pod \"model-serving-api-54d89bd5fc-7rtlr\" (UID: \"11e4d234-8c3e-4ad2-8931-99af2824b387\") " pod="opendatahub/model-serving-api-54d89bd5fc-7rtlr"
Sep 10 16:22:46.342704 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.342665    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-n9ffr"
Sep 10 16:22:46.399220 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.398678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e4d234-8c3e-4ad2-8931-99af2824b387-tls-certs\") pod \"model-serving-api-54d89bd5fc-7rtlr\" (UID: \"11e4d234-8c3e-4ad2-8931-99af2824b387\") " pod="opendatahub/model-serving-api-54d89bd5fc-7rtlr"
Sep 10 16:22:46.399220 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.398761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw8mt\" (UniqueName: \"kubernetes.io/projected/11e4d234-8c3e-4ad2-8931-99af2824b387-kube-api-access-sw8mt\") pod \"model-serving-api-54d89bd5fc-7rtlr\" (UID: \"11e4d234-8c3e-4ad2-8931-99af2824b387\") " pod="opendatahub/model-serving-api-54d89bd5fc-7rtlr"
Sep 10 16:22:46.399220 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:22:46.399075    2580 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 10 16:22:46.399220 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:22:46.399163    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11e4d234-8c3e-4ad2-8931-99af2824b387-tls-certs podName:11e4d234-8c3e-4ad2-8931-99af2824b387 nodeName:}" failed. No retries permitted until 2026-09-10 16:22:46.899124409 +0000 UTC m=+325.100166799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/11e4d234-8c3e-4ad2-8931-99af2824b387-tls-certs") pod "model-serving-api-54d89bd5fc-7rtlr" (UID: "11e4d234-8c3e-4ad2-8931-99af2824b387") : secret "model-serving-api-tls" not found
Sep 10 16:22:46.407965 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.407937    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw8mt\" (UniqueName: \"kubernetes.io/projected/11e4d234-8c3e-4ad2-8931-99af2824b387-kube-api-access-sw8mt\") pod \"model-serving-api-54d89bd5fc-7rtlr\" (UID: \"11e4d234-8c3e-4ad2-8931-99af2824b387\") " pod="opendatahub/model-serving-api-54d89bd5fc-7rtlr"
Sep 10 16:22:46.614387 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.614328    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n9ffr"]
Sep 10 16:22:46.617311 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:22:46.617285    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f7f3f3_6711_4eee_9ef1_34a8100a357f.slice/crio-5eae668fd97d8674054dc52b43830986f75c6f01f45a3bf16efa3254e581d526 WatchSource:0}: Error finding container 5eae668fd97d8674054dc52b43830986f75c6f01f45a3bf16efa3254e581d526: Status 404 returned error can't find the container with id 5eae668fd97d8674054dc52b43830986f75c6f01f45a3bf16efa3254e581d526
Sep 10 16:22:46.902785 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.902719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e4d234-8c3e-4ad2-8931-99af2824b387-tls-certs\") pod \"model-serving-api-54d89bd5fc-7rtlr\" (UID: \"11e4d234-8c3e-4ad2-8931-99af2824b387\") " pod="opendatahub/model-serving-api-54d89bd5fc-7rtlr"
Sep 10 16:22:46.905067 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:46.905045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e4d234-8c3e-4ad2-8931-99af2824b387-tls-certs\") pod \"model-serving-api-54d89bd5fc-7rtlr\" (UID: \"11e4d234-8c3e-4ad2-8931-99af2824b387\") " pod="opendatahub/model-serving-api-54d89bd5fc-7rtlr"
Sep 10 16:22:47.109538 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:47.109511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7rtlr"
Sep 10 16:22:47.379076 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:47.379047    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7rtlr"]
Sep 10 16:22:47.381719 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:22:47.381686    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e4d234_8c3e_4ad2_8931_99af2824b387.slice/crio-d3511e886012cd690b912ac5924b7253a38aad131acd538d89e618165414be08 WatchSource:0}: Error finding container d3511e886012cd690b912ac5924b7253a38aad131acd538d89e618165414be08: Status 404 returned error can't find the container with id d3511e886012cd690b912ac5924b7253a38aad131acd538d89e618165414be08
Sep 10 16:22:47.605831 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:47.605791    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-n9ffr" event={"ID":"14f7f3f3-6711-4eee-9ef1-34a8100a357f","Type":"ContainerStarted","Data":"5eae668fd97d8674054dc52b43830986f75c6f01f45a3bf16efa3254e581d526"}
Sep 10 16:22:47.607261 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:47.607228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7rtlr" event={"ID":"11e4d234-8c3e-4ad2-8931-99af2824b387","Type":"ContainerStarted","Data":"d3511e886012cd690b912ac5924b7253a38aad131acd538d89e618165414be08"}
Sep 10 16:22:50.619222 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:50.619185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7rtlr" event={"ID":"11e4d234-8c3e-4ad2-8931-99af2824b387","Type":"ContainerStarted","Data":"52354af4902d6fdcc93ba8ef4954dcde4ba9830631161caa92fd496907c680e5"}
Sep 10 16:22:50.619639 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:50.619243    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-7rtlr"
Sep 10 16:22:50.620669 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:50.620638    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-n9ffr" event={"ID":"14f7f3f3-6711-4eee-9ef1-34a8100a357f","Type":"ContainerStarted","Data":"e8adb85505aa8e98db954c0beb5737ae602b8f8ecc06e7a88f36d9367807bb01"}
Sep 10 16:22:50.620775 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:50.620763    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-n9ffr"
Sep 10 16:22:50.662374 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:50.662300    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-n9ffr" podStartSLOduration=2.396107806 podStartE2EDuration="5.662286427s" totalImagesPullingTime="3.266178621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:22:50.660881818 +0000 UTC m=+328.861924229" watchObservedRunningTime="2026-09-10 16:22:50.662286427 +0000 UTC m=+328.863328840"
Sep 10 16:22:50.662521 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:50.662414    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-7rtlr" podStartSLOduration=2.165802733 podStartE2EDuration="4.662406165s" totalImagesPullingTime="2.496603432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:22:50.639277873 +0000 UTC m=+328.840320283" watchObservedRunningTime="2026-09-10 16:22:50.662406165 +0000 UTC m=+328.863448577"
Sep 10 16:22:53.509874 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.509842    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"]
Sep 10 16:22:53.511888 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.511873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:53.518614 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.518576    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 16:22:53.518614 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.518607    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 16:22:53.518833 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.518641    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 16:22:53.518833 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.518759    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qsxzr\""
Sep 10 16:22:53.518833 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.518819    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 16:22:53.518989 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.518893    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 16:22:53.525110 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.525084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"]
Sep 10 16:22:53.549021 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.548995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc1812ef-b56a-4e01-828b-c58850fad789-metrics-cert\") pod \"lws-controller-manager-575575bffc-xp7vh\" (UID: \"cc1812ef-b56a-4e01-828b-c58850fad789\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:53.549110 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.549027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7fhq\" (UniqueName: \"kubernetes.io/projected/cc1812ef-b56a-4e01-828b-c58850fad789-kube-api-access-m7fhq\") pod \"lws-controller-manager-575575bffc-xp7vh\" (UID: \"cc1812ef-b56a-4e01-828b-c58850fad789\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:53.549110 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.549062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cc1812ef-b56a-4e01-828b-c58850fad789-manager-config\") pod \"lws-controller-manager-575575bffc-xp7vh\" (UID: \"cc1812ef-b56a-4e01-828b-c58850fad789\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:53.549194 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.549114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc1812ef-b56a-4e01-828b-c58850fad789-cert\") pod \"lws-controller-manager-575575bffc-xp7vh\" (UID: \"cc1812ef-b56a-4e01-828b-c58850fad789\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:53.650162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.650138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc1812ef-b56a-4e01-828b-c58850fad789-metrics-cert\") pod \"lws-controller-manager-575575bffc-xp7vh\" (UID: \"cc1812ef-b56a-4e01-828b-c58850fad789\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:53.650268 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.650167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fhq\" (UniqueName: \"kubernetes.io/projected/cc1812ef-b56a-4e01-828b-c58850fad789-kube-api-access-m7fhq\") pod \"lws-controller-manager-575575bffc-xp7vh\" (UID: \"cc1812ef-b56a-4e01-828b-c58850fad789\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:53.650268 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.650196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cc1812ef-b56a-4e01-828b-c58850fad789-manager-config\") pod \"lws-controller-manager-575575bffc-xp7vh\" (UID: \"cc1812ef-b56a-4e01-828b-c58850fad789\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:53.650268 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.650243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc1812ef-b56a-4e01-828b-c58850fad789-cert\") pod \"lws-controller-manager-575575bffc-xp7vh\" (UID: \"cc1812ef-b56a-4e01-828b-c58850fad789\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:53.650796 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.650775    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cc1812ef-b56a-4e01-828b-c58850fad789-manager-config\") pod \"lws-controller-manager-575575bffc-xp7vh\" (UID: \"cc1812ef-b56a-4e01-828b-c58850fad789\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:53.652639 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.652612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc1812ef-b56a-4e01-828b-c58850fad789-metrics-cert\") pod \"lws-controller-manager-575575bffc-xp7vh\" (UID: \"cc1812ef-b56a-4e01-828b-c58850fad789\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:53.652724 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.652670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc1812ef-b56a-4e01-828b-c58850fad789-cert\") pod \"lws-controller-manager-575575bffc-xp7vh\" (UID: \"cc1812ef-b56a-4e01-828b-c58850fad789\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:53.667548 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.667512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7fhq\" (UniqueName: \"kubernetes.io/projected/cc1812ef-b56a-4e01-828b-c58850fad789-kube-api-access-m7fhq\") pod \"lws-controller-manager-575575bffc-xp7vh\" (UID: \"cc1812ef-b56a-4e01-828b-c58850fad789\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:53.826137 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:53.826078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:54.111135 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:54.111110    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"]
Sep 10 16:22:54.113385 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:22:54.113357    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1812ef_b56a_4e01_828b_c58850fad789.slice/crio-4cc89d089044dde98e8b990107a463483179ddae5d4b62668aadaac4a280140d WatchSource:0}: Error finding container 4cc89d089044dde98e8b990107a463483179ddae5d4b62668aadaac4a280140d: Status 404 returned error can't find the container with id 4cc89d089044dde98e8b990107a463483179ddae5d4b62668aadaac4a280140d
Sep 10 16:22:54.636399 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:54.636356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh" event={"ID":"cc1812ef-b56a-4e01-828b-c58850fad789","Type":"ContainerStarted","Data":"4cc89d089044dde98e8b990107a463483179ddae5d4b62668aadaac4a280140d"}
Sep 10 16:22:56.645106 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:56.645071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh" event={"ID":"cc1812ef-b56a-4e01-828b-c58850fad789","Type":"ContainerStarted","Data":"6cf7843192af7f20ba4bcc6c778376458309ff03719346391b0340229a4ac784"}
Sep 10 16:22:56.645475 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:56.645139    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:22:56.696356 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:22:56.696308    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh" podStartSLOduration=1.5618563189999999 podStartE2EDuration="3.696293654s" totalImagesPullingTime="2.134437335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:22:56.696016201 +0000 UTC m=+334.897058613" watchObservedRunningTime="2026-09-10 16:22:56.696293654 +0000 UTC m=+334.897336065"
Sep 10 16:23:01.625948 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:23:01.625921    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-n9ffr"
Sep 10 16:23:01.628132 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:23:01.628107    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-7rtlr"
Sep 10 16:23:07.651081 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:23:07.651050    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-575575bffc-xp7vh"
Sep 10 16:25:13.541944 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:13.541908    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-grptb"]
Sep 10 16:25:13.544269 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:13.544249    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-grptb"
Sep 10 16:25:13.546760 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:13.546738    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mrgst\""
Sep 10 16:25:13.547390 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:13.547373    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 16:25:13.547488 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:13.547467    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 16:25:13.552642 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:13.552623    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-grptb"]
Sep 10 16:25:13.608174 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:13.608148    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rrfc\" (UniqueName: \"kubernetes.io/projected/ed34bc8c-2921-4543-92cb-2d2cd662c00f-kube-api-access-6rrfc\") pod \"authorino-59c974bd48-grptb\" (UID: \"ed34bc8c-2921-4543-92cb-2d2cd662c00f\") " pod="kuadrant-system/authorino-59c974bd48-grptb"
Sep 10 16:25:13.709184 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:13.709149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rrfc\" (UniqueName: \"kubernetes.io/projected/ed34bc8c-2921-4543-92cb-2d2cd662c00f-kube-api-access-6rrfc\") pod \"authorino-59c974bd48-grptb\" (UID: \"ed34bc8c-2921-4543-92cb-2d2cd662c00f\") " pod="kuadrant-system/authorino-59c974bd48-grptb"
Sep 10 16:25:13.717807 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:13.717780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rrfc\" (UniqueName: \"kubernetes.io/projected/ed34bc8c-2921-4543-92cb-2d2cd662c00f-kube-api-access-6rrfc\") pod \"authorino-59c974bd48-grptb\" (UID: \"ed34bc8c-2921-4543-92cb-2d2cd662c00f\") " pod="kuadrant-system/authorino-59c974bd48-grptb"
Sep 10 16:25:13.854016 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:13.853954    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-grptb"
Sep 10 16:25:14.120076 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:14.120054    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-grptb"]
Sep 10 16:25:14.122759 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:25:14.122735    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded34bc8c_2921_4543_92cb_2d2cd662c00f.slice/crio-00d6d4cad8459869ce452fe474fc31d617452086d04d993bff714f01f562a261 WatchSource:0}: Error finding container 00d6d4cad8459869ce452fe474fc31d617452086d04d993bff714f01f562a261: Status 404 returned error can't find the container with id 00d6d4cad8459869ce452fe474fc31d617452086d04d993bff714f01f562a261
Sep 10 16:25:15.127886 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:15.127840    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-grptb" event={"ID":"ed34bc8c-2921-4543-92cb-2d2cd662c00f","Type":"ContainerStarted","Data":"00d6d4cad8459869ce452fe474fc31d617452086d04d993bff714f01f562a261"}
Sep 10 16:25:17.139238 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:17.139201    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-grptb" event={"ID":"ed34bc8c-2921-4543-92cb-2d2cd662c00f","Type":"ContainerStarted","Data":"6af44d9aa7c2bbb718dc294e82a9ccf6bdb04364688c9823719dec3c85540611"}
Sep 10 16:25:17.163312 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:17.163268    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-grptb" podStartSLOduration=1.594925068 podStartE2EDuration="4.163229515s" totalImagesPullingTime="2.568304447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:25:17.161555755 +0000 UTC m=+475.362598157" watchObservedRunningTime="2026-09-10 16:25:17.163229515 +0000 UTC m=+475.364271925"
Sep 10 16:25:43.858203 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:43.858166    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-grptb"]
Sep 10 16:25:43.858799 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:43.858380    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-grptb" podUID="ed34bc8c-2921-4543-92cb-2d2cd662c00f" containerName="authorino" containerID="cri-o://6af44d9aa7c2bbb718dc294e82a9ccf6bdb04364688c9823719dec3c85540611" gracePeriod=30
Sep 10 16:25:44.092905 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:44.092879    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-grptb"
Sep 10 16:25:44.221217 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:44.221150    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rrfc\" (UniqueName: \"kubernetes.io/projected/ed34bc8c-2921-4543-92cb-2d2cd662c00f-kube-api-access-6rrfc\") pod \"ed34bc8c-2921-4543-92cb-2d2cd662c00f\" (UID: \"ed34bc8c-2921-4543-92cb-2d2cd662c00f\") "
Sep 10 16:25:44.223214 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:44.223191    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed34bc8c-2921-4543-92cb-2d2cd662c00f-kube-api-access-6rrfc" (OuterVolumeSpecName: "kube-api-access-6rrfc") pod "ed34bc8c-2921-4543-92cb-2d2cd662c00f" (UID: "ed34bc8c-2921-4543-92cb-2d2cd662c00f"). InnerVolumeSpecName "kube-api-access-6rrfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:25:44.244562 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:44.244526    2580 generic.go:356] "Generic (PLEG): container finished" podID="ed34bc8c-2921-4543-92cb-2d2cd662c00f" containerID="6af44d9aa7c2bbb718dc294e82a9ccf6bdb04364688c9823719dec3c85540611" exitCode=0
Sep 10 16:25:44.244654 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:44.244571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-grptb" event={"ID":"ed34bc8c-2921-4543-92cb-2d2cd662c00f","Type":"ContainerDied","Data":"6af44d9aa7c2bbb718dc294e82a9ccf6bdb04364688c9823719dec3c85540611"}
Sep 10 16:25:44.244654 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:44.244586    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-grptb"
Sep 10 16:25:44.244654 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:44.244617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-grptb" event={"ID":"ed34bc8c-2921-4543-92cb-2d2cd662c00f","Type":"ContainerDied","Data":"00d6d4cad8459869ce452fe474fc31d617452086d04d993bff714f01f562a261"}
Sep 10 16:25:44.244654 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:44.244635    2580 scope.go:117] "RemoveContainer" containerID="6af44d9aa7c2bbb718dc294e82a9ccf6bdb04364688c9823719dec3c85540611"
Sep 10 16:25:44.252589 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:44.252576    2580 scope.go:117] "RemoveContainer" containerID="6af44d9aa7c2bbb718dc294e82a9ccf6bdb04364688c9823719dec3c85540611"
Sep 10 16:25:44.252856 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:25:44.252829    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af44d9aa7c2bbb718dc294e82a9ccf6bdb04364688c9823719dec3c85540611\": container with ID starting with 6af44d9aa7c2bbb718dc294e82a9ccf6bdb04364688c9823719dec3c85540611 not found: ID does not exist" containerID="6af44d9aa7c2bbb718dc294e82a9ccf6bdb04364688c9823719dec3c85540611"
Sep 10 16:25:44.252948 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:44.252864    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af44d9aa7c2bbb718dc294e82a9ccf6bdb04364688c9823719dec3c85540611"} err="failed to get container status \"6af44d9aa7c2bbb718dc294e82a9ccf6bdb04364688c9823719dec3c85540611\": rpc error: code = NotFound desc = could not find container \"6af44d9aa7c2bbb718dc294e82a9ccf6bdb04364688c9823719dec3c85540611\": container with ID starting with 6af44d9aa7c2bbb718dc294e82a9ccf6bdb04364688c9823719dec3c85540611 not found: ID does not exist"
Sep 10 16:25:44.273816 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:44.273797    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-grptb"]
Sep 10 16:25:44.282332 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:44.282312    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-grptb"]
Sep 10 16:25:44.322515 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:44.322490    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rrfc\" (UniqueName: \"kubernetes.io/projected/ed34bc8c-2921-4543-92cb-2d2cd662c00f-kube-api-access-6rrfc\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:25:46.253135 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:25:46.253105    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed34bc8c-2921-4543-92cb-2d2cd662c00f" path="/var/lib/kubelet/pods/ed34bc8c-2921-4543-92cb-2d2cd662c00f/volumes"
Sep 10 16:26:27.509940 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.509906    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mr4zw"]
Sep 10 16:26:27.510317 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.510165    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed34bc8c-2921-4543-92cb-2d2cd662c00f" containerName="authorino"
Sep 10 16:26:27.510317 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.510175    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed34bc8c-2921-4543-92cb-2d2cd662c00f" containerName="authorino"
Sep 10 16:26:27.510317 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.510225    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed34bc8c-2921-4543-92cb-2d2cd662c00f" containerName="authorino"
Sep 10 16:26:27.513239 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.513223    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-mr4zw"
Sep 10 16:26:27.516253 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.516227    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-95vl7\""
Sep 10 16:26:27.516387 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.516324    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 16:26:27.529655 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.529636    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mr4zw"]
Sep 10 16:26:27.617117 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.617090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6605008-787d-480d-b24b-5e343dfe6d4d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mr4zw\" (UID: \"d6605008-787d-480d-b24b-5e343dfe6d4d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mr4zw"
Sep 10 16:26:27.617232 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.617126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-259d2\" (UniqueName: \"kubernetes.io/projected/d6605008-787d-480d-b24b-5e343dfe6d4d-kube-api-access-259d2\") pod \"payload-processing-5b54b7fbb7-mr4zw\" (UID: \"d6605008-787d-480d-b24b-5e343dfe6d4d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mr4zw"
Sep 10 16:26:27.718295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.718266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6605008-787d-480d-b24b-5e343dfe6d4d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mr4zw\" (UID: \"d6605008-787d-480d-b24b-5e343dfe6d4d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mr4zw"
Sep 10 16:26:27.718401 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.718299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-259d2\" (UniqueName: \"kubernetes.io/projected/d6605008-787d-480d-b24b-5e343dfe6d4d-kube-api-access-259d2\") pod \"payload-processing-5b54b7fbb7-mr4zw\" (UID: \"d6605008-787d-480d-b24b-5e343dfe6d4d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mr4zw"
Sep 10 16:26:27.718957 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.718939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6605008-787d-480d-b24b-5e343dfe6d4d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mr4zw\" (UID: \"d6605008-787d-480d-b24b-5e343dfe6d4d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mr4zw"
Sep 10 16:26:27.735884 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.735857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-259d2\" (UniqueName: \"kubernetes.io/projected/d6605008-787d-480d-b24b-5e343dfe6d4d-kube-api-access-259d2\") pod \"payload-processing-5b54b7fbb7-mr4zw\" (UID: \"d6605008-787d-480d-b24b-5e343dfe6d4d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mr4zw"
Sep 10 16:26:27.822320 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:27.822266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-mr4zw"
Sep 10 16:26:28.089352 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:28.089328    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mr4zw"]
Sep 10 16:26:28.091791 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:26:28.091764    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6605008_787d_480d_b24b_5e343dfe6d4d.slice/crio-00740ebac2bfbeb6d277cb26de8c24519fcde362d7bf806323b234e976668bd0 WatchSource:0}: Error finding container 00740ebac2bfbeb6d277cb26de8c24519fcde362d7bf806323b234e976668bd0: Status 404 returned error can't find the container with id 00740ebac2bfbeb6d277cb26de8c24519fcde362d7bf806323b234e976668bd0
Sep 10 16:26:28.411024 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:28.410943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-mr4zw" event={"ID":"d6605008-787d-480d-b24b-5e343dfe6d4d","Type":"ContainerStarted","Data":"00740ebac2bfbeb6d277cb26de8c24519fcde362d7bf806323b234e976668bd0"}
Sep 10 16:26:31.423917 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:31.423883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-mr4zw" event={"ID":"d6605008-787d-480d-b24b-5e343dfe6d4d","Type":"ContainerStarted","Data":"cbc5cbaf31a06920753c037271f140fe238faffab7f409fef162992119623980"}
Sep 10 16:26:31.424262 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:31.423978    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-mr4zw"
Sep 10 16:26:31.445385 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:31.445344    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-mr4zw" podStartSLOduration=2.111677221 podStartE2EDuration="4.445331438s" totalImagesPullingTime="2.333654217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:26:31.442607236 +0000 UTC m=+549.643649648" watchObservedRunningTime="2026-09-10 16:26:31.445331438 +0000 UTC m=+549.646373848"
Sep 10 16:26:42.429696 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:42.429664    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-mr4zw"
Sep 10 16:26:44.964415 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:44.964383    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"]
Sep 10 16:26:44.967645 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:44.967628    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:44.970754 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:44.970704    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 10 16:26:44.970878 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:44.970749    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 16:26:44.970878 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:44.970750    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 16:26:44.971000 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:44.970882    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h86xf\""
Sep 10 16:26:44.980178 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:44.980158    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"]
Sep 10 16:26:45.143235 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.143195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b390380-da50-4968-ae32-e69721f82605-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.143235 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.143239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b390380-da50-4968-ae32-e69721f82605-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.143452 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.143273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b390380-da50-4968-ae32-e69721f82605-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.143452 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.143308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b390380-da50-4968-ae32-e69721f82605-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.143452 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.143378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stfvg\" (UniqueName: \"kubernetes.io/projected/5b390380-da50-4968-ae32-e69721f82605-kube-api-access-stfvg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.143452 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.143410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b390380-da50-4968-ae32-e69721f82605-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.244652 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.244575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stfvg\" (UniqueName: \"kubernetes.io/projected/5b390380-da50-4968-ae32-e69721f82605-kube-api-access-stfvg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.244652 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.244624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b390380-da50-4968-ae32-e69721f82605-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.244825 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.244676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b390380-da50-4968-ae32-e69721f82605-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.244825 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.244704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b390380-da50-4968-ae32-e69721f82605-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.244825 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.244738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b390380-da50-4968-ae32-e69721f82605-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.244825 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.244776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b390380-da50-4968-ae32-e69721f82605-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.245107 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.245086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b390380-da50-4968-ae32-e69721f82605-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.245184 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.245163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b390380-da50-4968-ae32-e69721f82605-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.245229 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.245179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b390380-da50-4968-ae32-e69721f82605-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.246769 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.246744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b390380-da50-4968-ae32-e69721f82605-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.247056 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.247040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b390380-da50-4968-ae32-e69721f82605-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.253895 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.253875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stfvg\" (UniqueName: \"kubernetes.io/projected/5b390380-da50-4968-ae32-e69721f82605-kube-api-access-stfvg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz\" (UID: \"5b390380-da50-4968-ae32-e69721f82605\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.276672 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.276654    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:45.545335 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:45.545241    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"]
Sep 10 16:26:45.547491 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:26:45.547466    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b390380_da50_4968_ae32_e69721f82605.slice/crio-6d6a823ebfa290716416ffb8825528e86952000e0f8911861c6c6c6f3b05e2ea WatchSource:0}: Error finding container 6d6a823ebfa290716416ffb8825528e86952000e0f8911861c6c6c6f3b05e2ea: Status 404 returned error can't find the container with id 6d6a823ebfa290716416ffb8825528e86952000e0f8911861c6c6c6f3b05e2ea
Sep 10 16:26:46.484102 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:46.484056    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz" event={"ID":"5b390380-da50-4968-ae32-e69721f82605","Type":"ContainerStarted","Data":"6d6a823ebfa290716416ffb8825528e86952000e0f8911861c6c6c6f3b05e2ea"}
Sep 10 16:26:47.489492 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:47.489455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz" event={"ID":"5b390380-da50-4968-ae32-e69721f82605","Type":"ContainerStarted","Data":"b46ca9c0113de9dfb0b9ae96a8f39b525e8d2c8db8df0b30a721b73dabc0a144"}
Sep 10 16:26:47.510009 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:47.509959    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz" podStartSLOduration=2.350963805 podStartE2EDuration="3.50994336s" totalImagesPullingTime="1.158979555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:26:47.508835199 +0000 UTC m=+565.709877610" watchObservedRunningTime="2026-09-10 16:26:47.50994336 +0000 UTC m=+565.710985767"
Sep 10 16:26:55.277021 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:55.276986    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:55.277461 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:55.277040    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:55.289287 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:55.289264    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:26:55.530058 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:26:55.529987    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz"
Sep 10 16:27:10.230914 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.230876    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"]
Sep 10 16:27:10.238779 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.238755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.240905 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.240882    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 16:27:10.243903 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.243879    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"]
Sep 10 16:27:10.319495 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.319461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48xzf\" (UniqueName: \"kubernetes.io/projected/0f411e02-ca8f-4fad-b849-6008e9785c3c-kube-api-access-48xzf\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.319656 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.319514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f411e02-ca8f-4fad-b849-6008e9785c3c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.319656 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.319610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f411e02-ca8f-4fad-b849-6008e9785c3c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.319656 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.319635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f411e02-ca8f-4fad-b849-6008e9785c3c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.319851 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.319664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f411e02-ca8f-4fad-b849-6008e9785c3c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.319851 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.319714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f411e02-ca8f-4fad-b849-6008e9785c3c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.420495 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.420465    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f411e02-ca8f-4fad-b849-6008e9785c3c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.420495 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.420497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f411e02-ca8f-4fad-b849-6008e9785c3c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.420700 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.420521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f411e02-ca8f-4fad-b849-6008e9785c3c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.420700 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.420555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f411e02-ca8f-4fad-b849-6008e9785c3c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.420700 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.420585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48xzf\" (UniqueName: \"kubernetes.io/projected/0f411e02-ca8f-4fad-b849-6008e9785c3c-kube-api-access-48xzf\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.420700 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.420627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f411e02-ca8f-4fad-b849-6008e9785c3c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.420920 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.420896    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f411e02-ca8f-4fad-b849-6008e9785c3c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.420980 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.420956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f411e02-ca8f-4fad-b849-6008e9785c3c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.421061 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.421037    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f411e02-ca8f-4fad-b849-6008e9785c3c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.422683 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.422662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f411e02-ca8f-4fad-b849-6008e9785c3c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.422990 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.422972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f411e02-ca8f-4fad-b849-6008e9785c3c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.430022 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.430002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48xzf\" (UniqueName: \"kubernetes.io/projected/0f411e02-ca8f-4fad-b849-6008e9785c3c-kube-api-access-48xzf\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c\" (UID: \"0f411e02-ca8f-4fad-b849-6008e9785c3c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.549521 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.549493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:10.820198 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:10.820129    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"]
Sep 10 16:27:10.822292 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:27:10.822268    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f411e02_ca8f_4fad_b849_6008e9785c3c.slice/crio-57703f6764e2e331f53a9d2b3dd4a0f4bb2b846b0743ce43ea94a7d59d6232ef WatchSource:0}: Error finding container 57703f6764e2e331f53a9d2b3dd4a0f4bb2b846b0743ce43ea94a7d59d6232ef: Status 404 returned error can't find the container with id 57703f6764e2e331f53a9d2b3dd4a0f4bb2b846b0743ce43ea94a7d59d6232ef
Sep 10 16:27:11.577731 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:11.577696    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c" event={"ID":"0f411e02-ca8f-4fad-b849-6008e9785c3c","Type":"ContainerStarted","Data":"4bb622d7a3670df1440229506290a5b718aa54ff96f6b360a8ac81782d4a5b94"}
Sep 10 16:27:11.578076 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:11.577752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c" event={"ID":"0f411e02-ca8f-4fad-b849-6008e9785c3c","Type":"ContainerStarted","Data":"57703f6764e2e331f53a9d2b3dd4a0f4bb2b846b0743ce43ea94a7d59d6232ef"}
Sep 10 16:27:11.596903 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:11.596856    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c" podStartSLOduration=1.406645392 podStartE2EDuration="1.596842739s" totalImagesPullingTime="190.197347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:27:11.596335984 +0000 UTC m=+589.797378395" watchObservedRunningTime="2026-09-10 16:27:11.596842739 +0000 UTC m=+589.797885150"
Sep 10 16:27:20.550195 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:20.550162    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:20.550585 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:20.550228    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:20.562760 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:20.562735    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:20.627798 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:20.627769    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c"
Sep 10 16:27:49.057695 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.057614    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"]
Sep 10 16:27:49.061056 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.061033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.063594 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.063522    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 16:27:49.071611 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.071590    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"]
Sep 10 16:27:49.216571 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.216509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.216711 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.216590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.216711 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.216629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.216711 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.216705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.216846 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.216758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qp2z\" (UniqueName: \"kubernetes.io/projected/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-kube-api-access-7qp2z\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.216846 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.216785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.317286 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.317226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.317286 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.317263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.317441 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.317381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.317441 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.317423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.317570 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.317463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.317570 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.317487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qp2z\" (UniqueName: \"kubernetes.io/projected/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-kube-api-access-7qp2z\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.317682 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.317589    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.317682 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.317650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.317760 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.317741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.319555 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.319522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.319747 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.319729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.325806 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.325783    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qp2z\" (UniqueName: \"kubernetes.io/projected/b99d13c8-5ee1-4ea5-a38a-e2a297d8cced-kube-api-access-7qp2z\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4\" (UID: \"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.370195 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.370170    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:49.642275 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.642249    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"]
Sep 10 16:27:49.644407 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:27:49.644379    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb99d13c8_5ee1_4ea5_a38a_e2a297d8cced.slice/crio-146f53a6252a7c12800a23713af92827308e127caf01d4ed9a0311a50d2d7cd1 WatchSource:0}: Error finding container 146f53a6252a7c12800a23713af92827308e127caf01d4ed9a0311a50d2d7cd1: Status 404 returned error can't find the container with id 146f53a6252a7c12800a23713af92827308e127caf01d4ed9a0311a50d2d7cd1
Sep 10 16:27:49.646364 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.646347    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 16:27:49.727648 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:49.727620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4" event={"ID":"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced","Type":"ContainerStarted","Data":"146f53a6252a7c12800a23713af92827308e127caf01d4ed9a0311a50d2d7cd1"}
Sep 10 16:27:50.733306 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:50.733265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4" event={"ID":"b99d13c8-5ee1-4ea5-a38a-e2a297d8cced","Type":"ContainerStarted","Data":"00abd0e400e1d6607b157f84f3661a3da4251c1eb8625b70e798caaa4817775e"}
Sep 10 16:27:50.770733 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:50.770686    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4" podStartSLOduration=1.5967141329999999 podStartE2EDuration="1.770670366s" totalImagesPullingTime="173.956233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:27:50.770384195 +0000 UTC m=+628.971426607" watchObservedRunningTime="2026-09-10 16:27:50.770670366 +0000 UTC m=+628.971712777"
Sep 10 16:27:59.371032 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:59.371000    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:59.371461 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:59.371057    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:59.383011 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:59.382980    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:27:59.777552 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:27:59.777508    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4"
Sep 10 16:29:01.386328 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.386298    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d68676b9c-6k2dt"]
Sep 10 16:29:01.389356 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.389339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d68676b9c-6k2dt"
Sep 10 16:29:01.391853 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.391830    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 16:29:01.392454 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.392437    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 16:29:01.392511 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.392450    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 16:29:01.392575 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.392445    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mrgst\""
Sep 10 16:29:01.403584 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.403563    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d68676b9c-6k2dt"]
Sep 10 16:29:01.541984 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.541949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0969416-fcf6-4877-87f0-fe847a133951-tls-cert\") pod \"authorino-5d68676b9c-6k2dt\" (UID: \"b0969416-fcf6-4877-87f0-fe847a133951\") " pod="kuadrant-system/authorino-5d68676b9c-6k2dt"
Sep 10 16:29:01.542101 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.542006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8vjl\" (UniqueName: \"kubernetes.io/projected/b0969416-fcf6-4877-87f0-fe847a133951-kube-api-access-f8vjl\") pod \"authorino-5d68676b9c-6k2dt\" (UID: \"b0969416-fcf6-4877-87f0-fe847a133951\") " pod="kuadrant-system/authorino-5d68676b9c-6k2dt"
Sep 10 16:29:01.642738 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.642679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0969416-fcf6-4877-87f0-fe847a133951-tls-cert\") pod \"authorino-5d68676b9c-6k2dt\" (UID: \"b0969416-fcf6-4877-87f0-fe847a133951\") " pod="kuadrant-system/authorino-5d68676b9c-6k2dt"
Sep 10 16:29:01.642738 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.642717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8vjl\" (UniqueName: \"kubernetes.io/projected/b0969416-fcf6-4877-87f0-fe847a133951-kube-api-access-f8vjl\") pod \"authorino-5d68676b9c-6k2dt\" (UID: \"b0969416-fcf6-4877-87f0-fe847a133951\") " pod="kuadrant-system/authorino-5d68676b9c-6k2dt"
Sep 10 16:29:01.644928 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.644910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0969416-fcf6-4877-87f0-fe847a133951-tls-cert\") pod \"authorino-5d68676b9c-6k2dt\" (UID: \"b0969416-fcf6-4877-87f0-fe847a133951\") " pod="kuadrant-system/authorino-5d68676b9c-6k2dt"
Sep 10 16:29:01.651231 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.651211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8vjl\" (UniqueName: \"kubernetes.io/projected/b0969416-fcf6-4877-87f0-fe847a133951-kube-api-access-f8vjl\") pod \"authorino-5d68676b9c-6k2dt\" (UID: \"b0969416-fcf6-4877-87f0-fe847a133951\") " pod="kuadrant-system/authorino-5d68676b9c-6k2dt"
Sep 10 16:29:01.697822 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.697798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d68676b9c-6k2dt"
Sep 10 16:29:01.966225 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:01.966201    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d68676b9c-6k2dt"]
Sep 10 16:29:01.968715 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:29:01.968685    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0969416_fcf6_4877_87f0_fe847a133951.slice/crio-76ccbd059735c9586986839313bb80b72ddd9f3d26b6d9e0ffcc5f4fb268c26c WatchSource:0}: Error finding container 76ccbd059735c9586986839313bb80b72ddd9f3d26b6d9e0ffcc5f4fb268c26c: Status 404 returned error can't find the container with id 76ccbd059735c9586986839313bb80b72ddd9f3d26b6d9e0ffcc5f4fb268c26c
Sep 10 16:29:02.015565 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:02.015518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d68676b9c-6k2dt" event={"ID":"b0969416-fcf6-4877-87f0-fe847a133951","Type":"ContainerStarted","Data":"76ccbd059735c9586986839313bb80b72ddd9f3d26b6d9e0ffcc5f4fb268c26c"}
Sep 10 16:29:04.025207 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:04.025165    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d68676b9c-6k2dt" event={"ID":"b0969416-fcf6-4877-87f0-fe847a133951","Type":"ContainerStarted","Data":"1e241a3bba0ba542f8daffdc0cf43a1d11035a2088d77fd8070df77f8164627a"}
Sep 10 16:29:04.050483 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:04.050429    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d68676b9c-6k2dt" podStartSLOduration=1.971990203 podStartE2EDuration="3.050413281s" totalImagesPullingTime="1.078423078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:04.0488811 +0000 UTC m=+702.249923514" watchObservedRunningTime="2026-09-10 16:29:04.050413281 +0000 UTC m=+702.251455724"
Sep 10 16:29:29.014571 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.014480    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"]
Sep 10 16:29:29.018051 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.018032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"
Sep 10 16:29:29.021057 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.021038    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1abdaf-dockercfg-8x9vz\""
Sep 10 16:29:29.021684 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.021661    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1abdaf\""
Sep 10 16:29:29.031606 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.031587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"]
Sep 10 16:29:29.035976 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.035956    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"]
Sep 10 16:29:29.039684 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.039667    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"
Sep 10 16:29:29.051717 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.051686    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"]
Sep 10 16:29:29.158108 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.158079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9sw9\" (UniqueName: \"kubernetes.io/projected/ad4c5804-56fa-43ad-a804-82a72acb32b3-kube-api-access-f9sw9\") pod \"payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb\" (UID: \"ad4c5804-56fa-43ad-a804-82a72acb32b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"
Sep 10 16:29:29.158235 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.158119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25nhf\" (UniqueName: \"kubernetes.io/projected/322d822c-ffba-4a59-a920-64079c784880-kube-api-access-25nhf\") pod \"payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk\" (UID: \"322d822c-ffba-4a59-a920-64079c784880\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"
Sep 10 16:29:29.158235 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.158209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322d822c-ffba-4a59-a920-64079c784880-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk\" (UID: \"322d822c-ffba-4a59-a920-64079c784880\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"
Sep 10 16:29:29.158312 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.158243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4c5804-56fa-43ad-a804-82a72acb32b3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb\" (UID: \"ad4c5804-56fa-43ad-a804-82a72acb32b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"
Sep 10 16:29:29.258558 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.258522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322d822c-ffba-4a59-a920-64079c784880-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk\" (UID: \"322d822c-ffba-4a59-a920-64079c784880\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"
Sep 10 16:29:29.258669 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.258572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4c5804-56fa-43ad-a804-82a72acb32b3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb\" (UID: \"ad4c5804-56fa-43ad-a804-82a72acb32b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"
Sep 10 16:29:29.258669 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.258607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9sw9\" (UniqueName: \"kubernetes.io/projected/ad4c5804-56fa-43ad-a804-82a72acb32b3-kube-api-access-f9sw9\") pod \"payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb\" (UID: \"ad4c5804-56fa-43ad-a804-82a72acb32b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"
Sep 10 16:29:29.258669 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.258630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25nhf\" (UniqueName: \"kubernetes.io/projected/322d822c-ffba-4a59-a920-64079c784880-kube-api-access-25nhf\") pod \"payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk\" (UID: \"322d822c-ffba-4a59-a920-64079c784880\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"
Sep 10 16:29:29.259209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.259189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322d822c-ffba-4a59-a920-64079c784880-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk\" (UID: \"322d822c-ffba-4a59-a920-64079c784880\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"
Sep 10 16:29:29.259209 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.259202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4c5804-56fa-43ad-a804-82a72acb32b3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb\" (UID: \"ad4c5804-56fa-43ad-a804-82a72acb32b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"
Sep 10 16:29:29.267696 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.267645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25nhf\" (UniqueName: \"kubernetes.io/projected/322d822c-ffba-4a59-a920-64079c784880-kube-api-access-25nhf\") pod \"payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk\" (UID: \"322d822c-ffba-4a59-a920-64079c784880\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"
Sep 10 16:29:29.267782 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.267696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9sw9\" (UniqueName: \"kubernetes.io/projected/ad4c5804-56fa-43ad-a804-82a72acb32b3-kube-api-access-f9sw9\") pod \"payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb\" (UID: \"ad4c5804-56fa-43ad-a804-82a72acb32b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"
Sep 10 16:29:29.327296 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.327277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"
Sep 10 16:29:29.349034 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.349014    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"
Sep 10 16:29:29.613805 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.613777    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"]
Sep 10 16:29:29.617009 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:29:29.616978    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322d822c_ffba_4a59_a920_64079c784880.slice/crio-3ca6f323be0ec61b9058082eae46dd41280f04918f700bbef0ede9932ad5ab30 WatchSource:0}: Error finding container 3ca6f323be0ec61b9058082eae46dd41280f04918f700bbef0ede9932ad5ab30: Status 404 returned error can't find the container with id 3ca6f323be0ec61b9058082eae46dd41280f04918f700bbef0ede9932ad5ab30
Sep 10 16:29:29.635817 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:29.635796    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"]
Sep 10 16:29:29.649546 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:29:29.649495    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4c5804_56fa_43ad_a804_82a72acb32b3.slice/crio-13cd1487bac57cd50deeef5d2e603df1cdf16325b472e28a59e746fe3e057b36 WatchSource:0}: Error finding container 13cd1487bac57cd50deeef5d2e603df1cdf16325b472e28a59e746fe3e057b36: Status 404 returned error can't find the container with id 13cd1487bac57cd50deeef5d2e603df1cdf16325b472e28a59e746fe3e057b36
Sep 10 16:29:30.132886 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:30.132843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb" event={"ID":"ad4c5804-56fa-43ad-a804-82a72acb32b3","Type":"ContainerStarted","Data":"2035c64db9fa08979e1feae139fce96aeef06591f4a911283f552344d607c3d7"}
Sep 10 16:29:30.133333 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:30.132907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb" event={"ID":"ad4c5804-56fa-43ad-a804-82a72acb32b3","Type":"ContainerStarted","Data":"13cd1487bac57cd50deeef5d2e603df1cdf16325b472e28a59e746fe3e057b36"}
Sep 10 16:29:30.133333 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:30.133022    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"
Sep 10 16:29:30.134417 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:30.134399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk" event={"ID":"322d822c-ffba-4a59-a920-64079c784880","Type":"ContainerStarted","Data":"ed74d70cfdcb245ba30954c99ea2b3afa0e7b83090b6f7cea731ecdeb559a329"}
Sep 10 16:29:30.134510 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:30.134434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk" event={"ID":"322d822c-ffba-4a59-a920-64079c784880","Type":"ContainerStarted","Data":"3ca6f323be0ec61b9058082eae46dd41280f04918f700bbef0ede9932ad5ab30"}
Sep 10 16:29:30.134580 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:30.134522    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"
Sep 10 16:29:30.155208 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:30.155162    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb" podStartSLOduration=1.155149443 podStartE2EDuration="1.155149443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:30.154625273 +0000 UTC m=+728.355667686" watchObservedRunningTime="2026-09-10 16:29:30.155149443 +0000 UTC m=+728.356191853"
Sep 10 16:29:30.178401 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:30.178357    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk" podStartSLOduration=2.178339989 podStartE2EDuration="2.178339989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:30.176327461 +0000 UTC m=+728.377369873" watchObservedRunningTime="2026-09-10 16:29:30.178339989 +0000 UTC m=+728.379382405"
Sep 10 16:29:35.749577 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:35.749541    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"]
Sep 10 16:29:35.753042 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:35.753019    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"
Sep 10 16:29:35.756007 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:35.755983    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-2d818c\""
Sep 10 16:29:35.756095 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:35.756028    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-2d818c-dockercfg-qqxj8\""
Sep 10 16:29:35.765376 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:35.765356    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"]
Sep 10 16:29:35.914926 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:35.914898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc71372-93fa-440a-bd6e-af84554d02a1-plugins-config-volume\") pod \"payload-processing-e2e-autores-2d818c-7b955d8589-hql4v\" (UID: \"5bc71372-93fa-440a-bd6e-af84554d02a1\") " pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"
Sep 10 16:29:35.915095 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:35.914949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkffx\" (UniqueName: \"kubernetes.io/projected/5bc71372-93fa-440a-bd6e-af84554d02a1-kube-api-access-mkffx\") pod \"payload-processing-e2e-autores-2d818c-7b955d8589-hql4v\" (UID: \"5bc71372-93fa-440a-bd6e-af84554d02a1\") " pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"
Sep 10 16:29:36.015873 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:36.015842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkffx\" (UniqueName: \"kubernetes.io/projected/5bc71372-93fa-440a-bd6e-af84554d02a1-kube-api-access-mkffx\") pod \"payload-processing-e2e-autores-2d818c-7b955d8589-hql4v\" (UID: \"5bc71372-93fa-440a-bd6e-af84554d02a1\") " pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"
Sep 10 16:29:36.015997 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:36.015919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc71372-93fa-440a-bd6e-af84554d02a1-plugins-config-volume\") pod \"payload-processing-e2e-autores-2d818c-7b955d8589-hql4v\" (UID: \"5bc71372-93fa-440a-bd6e-af84554d02a1\") " pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"
Sep 10 16:29:36.016564 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:36.016520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc71372-93fa-440a-bd6e-af84554d02a1-plugins-config-volume\") pod \"payload-processing-e2e-autores-2d818c-7b955d8589-hql4v\" (UID: \"5bc71372-93fa-440a-bd6e-af84554d02a1\") " pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"
Sep 10 16:29:36.023850 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:36.023828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkffx\" (UniqueName: \"kubernetes.io/projected/5bc71372-93fa-440a-bd6e-af84554d02a1-kube-api-access-mkffx\") pod \"payload-processing-e2e-autores-2d818c-7b955d8589-hql4v\" (UID: \"5bc71372-93fa-440a-bd6e-af84554d02a1\") " pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"
Sep 10 16:29:36.063207 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:36.063180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"
Sep 10 16:29:36.334502 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:36.334423    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"]
Sep 10 16:29:36.337704 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:29:36.337675    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc71372_93fa_440a_bd6e_af84554d02a1.slice/crio-fd35dab4ffdf9793be5b6e4ae4ef356643a56b0bab17dc7189f68c407c552d00 WatchSource:0}: Error finding container fd35dab4ffdf9793be5b6e4ae4ef356643a56b0bab17dc7189f68c407c552d00: Status 404 returned error can't find the container with id fd35dab4ffdf9793be5b6e4ae4ef356643a56b0bab17dc7189f68c407c552d00
Sep 10 16:29:37.167073 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:37.167033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v" event={"ID":"5bc71372-93fa-440a-bd6e-af84554d02a1","Type":"ContainerStarted","Data":"37b559913c10724c89bf578da7ab1b44033e9006768825a20c45733de3e2a654"}
Sep 10 16:29:37.167441 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:37.167135    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"
Sep 10 16:29:37.167441 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:37.167163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v" event={"ID":"5bc71372-93fa-440a-bd6e-af84554d02a1","Type":"ContainerStarted","Data":"fd35dab4ffdf9793be5b6e4ae4ef356643a56b0bab17dc7189f68c407c552d00"}
Sep 10 16:29:37.187562 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:37.187502    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v" podStartSLOduration=2.1874887100000002 podStartE2EDuration="2.18748871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:37.185179424 +0000 UTC m=+735.386221835" watchObservedRunningTime="2026-09-10 16:29:37.18748871 +0000 UTC m=+735.388531120"
Sep 10 16:29:38.875674 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:38.875644    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"]
Sep 10 16:29:38.878979 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:38.878963    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"
Sep 10 16:29:38.882196 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:38.882179    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-25d77f0b-dockercfg-2mqcg\""
Sep 10 16:29:38.882265 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:38.882207    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-25d77f0b\""
Sep 10 16:29:38.892123 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:38.892103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"]
Sep 10 16:29:38.940816 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:38.940787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl6h5\" (UniqueName: \"kubernetes.io/projected/0b164e16-6a0b-48e7-b1ef-f095b923cc28-kube-api-access-sl6h5\") pod \"payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt\" (UID: \"0b164e16-6a0b-48e7-b1ef-f095b923cc28\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"
Sep 10 16:29:38.940928 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:38.940835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b164e16-6a0b-48e7-b1ef-f095b923cc28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt\" (UID: \"0b164e16-6a0b-48e7-b1ef-f095b923cc28\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"
Sep 10 16:29:39.041843 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:39.041815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl6h5\" (UniqueName: \"kubernetes.io/projected/0b164e16-6a0b-48e7-b1ef-f095b923cc28-kube-api-access-sl6h5\") pod \"payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt\" (UID: \"0b164e16-6a0b-48e7-b1ef-f095b923cc28\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"
Sep 10 16:29:39.041957 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:39.041860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b164e16-6a0b-48e7-b1ef-f095b923cc28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt\" (UID: \"0b164e16-6a0b-48e7-b1ef-f095b923cc28\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"
Sep 10 16:29:39.042527 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:39.042510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b164e16-6a0b-48e7-b1ef-f095b923cc28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt\" (UID: \"0b164e16-6a0b-48e7-b1ef-f095b923cc28\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"
Sep 10 16:29:39.049984 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:39.049966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl6h5\" (UniqueName: \"kubernetes.io/projected/0b164e16-6a0b-48e7-b1ef-f095b923cc28-kube-api-access-sl6h5\") pod \"payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt\" (UID: \"0b164e16-6a0b-48e7-b1ef-f095b923cc28\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"
Sep 10 16:29:39.188854 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:39.188800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"
Sep 10 16:29:39.469055 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:39.469029    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"]
Sep 10 16:29:39.471062 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:29:39.471031    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b164e16_6a0b_48e7_b1ef_f095b923cc28.slice/crio-8f619f7d5b759bbb621b4b8abfe2f139b4c16c2405968ecdd15f3afea7dae20e WatchSource:0}: Error finding container 8f619f7d5b759bbb621b4b8abfe2f139b4c16c2405968ecdd15f3afea7dae20e: Status 404 returned error can't find the container with id 8f619f7d5b759bbb621b4b8abfe2f139b4c16c2405968ecdd15f3afea7dae20e
Sep 10 16:29:39.963919 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:39.963889    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"]
Sep 10 16:29:39.967098 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:39.967080    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:39.970159 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:29:39.970123    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" is forbidden: User \"system:node:ip-10-0-138-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-138-74.ec2.internal' and this object" podUID="ad232a58-d8e1-4a88-88b6-264dbd967238" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:39.970159 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:29:39.970132    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-conflict-a-a05a36b8-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-138-74.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-138-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"e2e-conflict-a-a05a36b8-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 10 16:29:39.981630 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:39.981609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"]
Sep 10 16:29:40.052786 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.052762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-dshm\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.052893 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.052797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msmpg\" (UniqueName: \"kubernetes.io/projected/ad232a58-d8e1-4a88-88b6-264dbd967238-kube-api-access-msmpg\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.052893 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.052824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-tmp-dir\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.052893 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.052841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad232a58-d8e1-4a88-88b6-264dbd967238-tls-certs\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.053006 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.052894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-model-cache\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.053006 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.052917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-home\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.153763 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.153738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-home\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.153869 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.153773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-dshm\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.153869 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.153794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msmpg\" (UniqueName: \"kubernetes.io/projected/ad232a58-d8e1-4a88-88b6-264dbd967238-kube-api-access-msmpg\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.153964 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.153926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-tmp-dir\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.154014 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.153972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad232a58-d8e1-4a88-88b6-264dbd967238-tls-certs\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.154075 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.154037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-model-cache\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.154125 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.154099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-home\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.154256 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.154237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-tmp-dir\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.154349 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.154332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-model-cache\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.155964 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.155946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-dshm\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.163349 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.163330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msmpg\" (UniqueName: \"kubernetes.io/projected/ad232a58-d8e1-4a88-88b6-264dbd967238-kube-api-access-msmpg\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:40.180207 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.180184    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt" event={"ID":"0b164e16-6a0b-48e7-b1ef-f095b923cc28","Type":"ContainerStarted","Data":"a6e46cf87eb4dd1de700f936f9efb95c8faefb900ffa6db47c6f8f9b875aae23"}
Sep 10 16:29:40.180272 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.180231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt" event={"ID":"0b164e16-6a0b-48e7-b1ef-f095b923cc28","Type":"ContainerStarted","Data":"8f619f7d5b759bbb621b4b8abfe2f139b4c16c2405968ecdd15f3afea7dae20e"}
Sep 10 16:29:40.180272 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.180261    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"
Sep 10 16:29:40.199431 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:40.199299    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt" podStartSLOduration=2.1992808090000002 podStartE2EDuration="2.199280809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:40.197584122 +0000 UTC m=+738.398626534" watchObservedRunningTime="2026-09-10 16:29:40.199280809 +0000 UTC m=+738.400323223"
Sep 10 16:29:41.137490 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:41.137456    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-a05a36b8-kserve-self-signed-certs\""
Sep 10 16:29:41.141115 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:41.141088    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"
Sep 10 16:29:41.141233 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:41.141153    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"
Sep 10 16:29:41.147035 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:41.147004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad232a58-d8e1-4a88-88b6-264dbd967238-tls-certs\") pod \"e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") " pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:41.175921 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:41.175896    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:41.456901 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:41.456861    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"]
Sep 10 16:29:41.459090 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:29:41.459060    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad232a58_d8e1_4a88_88b6_264dbd967238.slice/crio-819252658c7862b62ff57c8236e56216b0ab3a1f6f47d0adf163885436c6f918 WatchSource:0}: Error finding container 819252658c7862b62ff57c8236e56216b0ab3a1f6f47d0adf163885436c6f918: Status 404 returned error can't find the container with id 819252658c7862b62ff57c8236e56216b0ab3a1f6f47d0adf163885436c6f918
Sep 10 16:29:42.193401 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:42.193364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg" event={"ID":"ad232a58-d8e1-4a88-88b6-264dbd967238","Type":"ContainerStarted","Data":"1fe8a2f256488006b18f03ecb515b7225f34dc83e79512247e829d943b2931f4"}
Sep 10 16:29:42.193401 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:42.193403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg" event={"ID":"ad232a58-d8e1-4a88-88b6-264dbd967238","Type":"ContainerStarted","Data":"819252658c7862b62ff57c8236e56216b0ab3a1f6f47d0adf163885436c6f918"}
Sep 10 16:29:42.217970 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:42.217926    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg" podStartSLOduration=3.217913108 podStartE2EDuration="3.217913108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:42.216490545 +0000 UTC m=+740.417532956" watchObservedRunningTime="2026-09-10 16:29:42.217913108 +0000 UTC m=+740.418955518"
Sep 10 16:29:48.173720 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:48.173685    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"
Sep 10 16:29:50.218684 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.218648    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"]
Sep 10 16:29:50.222062 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.222046    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"
Sep 10 16:29:50.224700 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.224677    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9f6082\""
Sep 10 16:29:50.224700 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.224694    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9f6082-dockercfg-f2q4f\""
Sep 10 16:29:50.234818 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.234797    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"]
Sep 10 16:29:50.275219 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.275193    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"]
Sep 10 16:29:50.278234 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.278219    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"
Sep 10 16:29:50.294599 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.294580    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"]
Sep 10 16:29:50.342308 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.342284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e89656b6-d9d2-4cd1-b742-1ed8e2468f94-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9\" (UID: \"e89656b6-d9d2-4cd1-b742-1ed8e2468f94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"
Sep 10 16:29:50.342411 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.342329    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkx6l\" (UniqueName: \"kubernetes.io/projected/e89656b6-d9d2-4cd1-b742-1ed8e2468f94-kube-api-access-rkx6l\") pod \"payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9\" (UID: \"e89656b6-d9d2-4cd1-b742-1ed8e2468f94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"
Sep 10 16:29:50.342411 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.342387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6\" (UID: \"4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"
Sep 10 16:29:50.342485 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.342421    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbcm2\" (UniqueName: \"kubernetes.io/projected/4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3-kube-api-access-qbcm2\") pod \"payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6\" (UID: \"4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"
Sep 10 16:29:50.443450 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.443412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkx6l\" (UniqueName: \"kubernetes.io/projected/e89656b6-d9d2-4cd1-b742-1ed8e2468f94-kube-api-access-rkx6l\") pod \"payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9\" (UID: \"e89656b6-d9d2-4cd1-b742-1ed8e2468f94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"
Sep 10 16:29:50.443596 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.443465    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6\" (UID: \"4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"
Sep 10 16:29:50.443596 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.443485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbcm2\" (UniqueName: \"kubernetes.io/projected/4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3-kube-api-access-qbcm2\") pod \"payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6\" (UID: \"4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"
Sep 10 16:29:50.443596 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.443578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e89656b6-d9d2-4cd1-b742-1ed8e2468f94-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9\" (UID: \"e89656b6-d9d2-4cd1-b742-1ed8e2468f94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"
Sep 10 16:29:50.444187 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.444167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6\" (UID: \"4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"
Sep 10 16:29:50.444229 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.444211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e89656b6-d9d2-4cd1-b742-1ed8e2468f94-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9\" (UID: \"e89656b6-d9d2-4cd1-b742-1ed8e2468f94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"
Sep 10 16:29:50.452640 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.452622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkx6l\" (UniqueName: \"kubernetes.io/projected/e89656b6-d9d2-4cd1-b742-1ed8e2468f94-kube-api-access-rkx6l\") pod \"payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9\" (UID: \"e89656b6-d9d2-4cd1-b742-1ed8e2468f94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"
Sep 10 16:29:50.454072 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.454055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbcm2\" (UniqueName: \"kubernetes.io/projected/4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3-kube-api-access-qbcm2\") pod \"payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6\" (UID: \"4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"
Sep 10 16:29:50.531520 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.531499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"
Sep 10 16:29:50.588148 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.588056    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"
Sep 10 16:29:50.802463 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.802441    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"]
Sep 10 16:29:50.804004 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:29:50.803977    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afdc4f6_a5b8_4ad8_9d18_e4724edc61c3.slice/crio-d473335a57c4e85f9390ae9b84966395a342b33774ab2d7d27d94fcb86efb3bd WatchSource:0}: Error finding container d473335a57c4e85f9390ae9b84966395a342b33774ab2d7d27d94fcb86efb3bd: Status 404 returned error can't find the container with id d473335a57c4e85f9390ae9b84966395a342b33774ab2d7d27d94fcb86efb3bd
Sep 10 16:29:50.869426 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:50.869404    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"]
Sep 10 16:29:50.871291 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:29:50.871268    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89656b6_d9d2_4cd1_b742_1ed8e2468f94.slice/crio-68e3536a5aec4be76a3f87cec5df227b806541a72d8c4fa648f2ed16bf1908b8 WatchSource:0}: Error finding container 68e3536a5aec4be76a3f87cec5df227b806541a72d8c4fa648f2ed16bf1908b8: Status 404 returned error can't find the container with id 68e3536a5aec4be76a3f87cec5df227b806541a72d8c4fa648f2ed16bf1908b8
Sep 10 16:29:51.176671 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:51.176590    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:51.176671 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:51.176649    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:51.189166 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:51.189143    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"
Sep 10 16:29:51.195378 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:51.195355    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:51.239592 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:51.239556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9" event={"ID":"e89656b6-d9d2-4cd1-b742-1ed8e2468f94","Type":"ContainerStarted","Data":"9caa9a07e59d05aa0c5a269566b9a46350f4f6827d1171b6a7184c7202dda8c4"}
Sep 10 16:29:51.239982 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:51.239616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9" event={"ID":"e89656b6-d9d2-4cd1-b742-1ed8e2468f94","Type":"ContainerStarted","Data":"68e3536a5aec4be76a3f87cec5df227b806541a72d8c4fa648f2ed16bf1908b8"}
Sep 10 16:29:51.239982 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:51.239690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"
Sep 10 16:29:51.241422 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:51.241368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6" event={"ID":"4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3","Type":"ContainerStarted","Data":"1febb2a2974e4bae1c1523c3b875d67408771f8fe47933718aba8935ac979332"}
Sep 10 16:29:51.241422 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:51.241418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6" event={"ID":"4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3","Type":"ContainerStarted","Data":"d473335a57c4e85f9390ae9b84966395a342b33774ab2d7d27d94fcb86efb3bd"}
Sep 10 16:29:51.241630 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:51.241465    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"
Sep 10 16:29:51.253823 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:51.253803    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:29:51.262212 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:51.262170    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9" podStartSLOduration=1.262156684 podStartE2EDuration="1.262156684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:51.260593108 +0000 UTC m=+749.461635519" watchObservedRunningTime="2026-09-10 16:29:51.262156684 +0000 UTC m=+749.463199162"
Sep 10 16:29:51.291514 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:29:51.291474    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6" podStartSLOduration=1.291463848 podStartE2EDuration="1.291463848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:51.289849807 +0000 UTC m=+749.490892219" watchObservedRunningTime="2026-09-10 16:29:51.291463848 +0000 UTC m=+749.492506258"
Sep 10 16:30:00.145076 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.145036    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2d818c-29817630-7d9jn"]
Sep 10 16:30:00.148729 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.148705    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817630-7nbjs"]
Sep 10 16:30:00.148858 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.148809    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2d818c-29817630-7d9jn"
Sep 10 16:30:00.151103 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.151081    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\""
Sep 10 16:30:00.151744 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.151728    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1abdaf-29817630-8jgcc"]
Sep 10 16:30:00.151870 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.151853    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817630-7nbjs"
Sep 10 16:30:00.154683 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.154666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1abdaf-29817630-8jgcc"
Sep 10 16:30:00.160026 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.159933    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2d818c-29817630-7d9jn"
Sep 10 16:30:00.161903 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.161870    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817630-7nbjs"]
Sep 10 16:30:00.164315 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.164215    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2d818c-29817630-7d9jn"]
Sep 10 16:30:00.167979 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.167956    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1abdaf-29817630-8jgcc"]
Sep 10 16:30:00.170852 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.170833    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817630-7nbjs"
Sep 10 16:30:00.205301 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.205122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1abdaf-29817630-8jgcc"
Sep 10 16:30:00.441953 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.441926    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2d818c-29817630-7d9jn"]
Sep 10 16:30:00.443674 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:30:00.443649    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb636e147_633a_403b_9858_2df91ba32080.slice/crio-4b127b0c90e9cccb42898f2fefd4b068fbb91627c935525a642efd0a8d5390dd WatchSource:0}: Error finding container 4b127b0c90e9cccb42898f2fefd4b068fbb91627c935525a642efd0a8d5390dd: Status 404 returned error can't find the container with id 4b127b0c90e9cccb42898f2fefd4b068fbb91627c935525a642efd0a8d5390dd
Sep 10 16:30:00.459474 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.459455    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817630-7nbjs"]
Sep 10 16:30:00.460714 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:30:00.460687    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55dc7d49_aa0a_4869_8bb6_6d6bb94d1314.slice/crio-007eb4f98cb2fa498fe9ce7096d1cf4e84bcc7de0120d56fbf4dbc0360196541 WatchSource:0}: Error finding container 007eb4f98cb2fa498fe9ce7096d1cf4e84bcc7de0120d56fbf4dbc0360196541: Status 404 returned error can't find the container with id 007eb4f98cb2fa498fe9ce7096d1cf4e84bcc7de0120d56fbf4dbc0360196541
Sep 10 16:30:00.497496 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:00.497475    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1abdaf-29817630-8jgcc"]
Sep 10 16:30:00.499343 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:30:00.499317    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3a5461_0fcb_4eb1_a98a_bf5d3e1a84ab.slice/crio-7288f75cf5f8911efbdcd9703e49cc6e22638cd4a4a4660b390a84810ea5d2e7 WatchSource:0}: Error finding container 7288f75cf5f8911efbdcd9703e49cc6e22638cd4a4a4660b390a84810ea5d2e7: Status 404 returned error can't find the container with id 7288f75cf5f8911efbdcd9703e49cc6e22638cd4a4a4660b390a84810ea5d2e7
Sep 10 16:30:01.294845 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:01.294808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817630-7nbjs" event={"ID":"55dc7d49-aa0a-4869-8bb6-6d6bb94d1314","Type":"ContainerStarted","Data":"007eb4f98cb2fa498fe9ce7096d1cf4e84bcc7de0120d56fbf4dbc0360196541"}
Sep 10 16:30:01.296335 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:01.296304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1abdaf-29817630-8jgcc" event={"ID":"be3a5461-0fcb-4eb1-a98a-bf5d3e1a84ab","Type":"ContainerStarted","Data":"7288f75cf5f8911efbdcd9703e49cc6e22638cd4a4a4660b390a84810ea5d2e7"}
Sep 10 16:30:01.297780 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:01.297756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2d818c-29817630-7d9jn" event={"ID":"b636e147-633a-403b-9858-2df91ba32080","Type":"ContainerStarted","Data":"4b127b0c90e9cccb42898f2fefd4b068fbb91627c935525a642efd0a8d5390dd"}
Sep 10 16:30:02.262645 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:02.262612    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"
Sep 10 16:30:02.262825 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:02.262683    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"
Sep 10 16:30:03.310930 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:03.310896    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1abdaf-29817630-8jgcc" event={"ID":"be3a5461-0fcb-4eb1-a98a-bf5d3e1a84ab","Type":"ContainerStarted","Data":"6a7a87b5149b05c8b64ecc1d40e621af1fd1e0e93775a6f85c8f23402fc30216"}
Sep 10 16:30:03.312455 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:03.312426    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2d818c-29817630-7d9jn" event={"ID":"b636e147-633a-403b-9858-2df91ba32080","Type":"ContainerStarted","Data":"77441092c5a25b5e615d1bc18c96ef154fbc5ee47b279c1d70d7262959aad73a"}
Sep 10 16:30:03.331024 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:03.330974    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1abdaf-29817630-8jgcc" podStartSLOduration=1.2637492240000001 podStartE2EDuration="3.330958463s" totalImagesPullingTime="2.067209239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:03.329183808 +0000 UTC m=+761.530226219" watchObservedRunningTime="2026-09-10 16:30:03.330958463 +0000 UTC m=+761.532000874"
Sep 10 16:30:03.348683 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:03.348632    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2d818c-29817630-7d9jn" podStartSLOduration=1.221894207 podStartE2EDuration="3.348615771s" totalImagesPullingTime="2.126721564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:03.347485616 +0000 UTC m=+761.548528029" watchObservedRunningTime="2026-09-10 16:30:03.348615771 +0000 UTC m=+761.549658181"
Sep 10 16:30:04.318178 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:04.318142    2580 generic.go:356] "Generic (PLEG): container finished" podID="be3a5461-0fcb-4eb1-a98a-bf5d3e1a84ab" containerID="6a7a87b5149b05c8b64ecc1d40e621af1fd1e0e93775a6f85c8f23402fc30216" exitCode=0
Sep 10 16:30:04.318493 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:04.318223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1abdaf-29817630-8jgcc" event={"ID":"be3a5461-0fcb-4eb1-a98a-bf5d3e1a84ab","Type":"ContainerDied","Data":"6a7a87b5149b05c8b64ecc1d40e621af1fd1e0e93775a6f85c8f23402fc30216"}
Sep 10 16:30:04.319988 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:04.319966    2580 generic.go:356] "Generic (PLEG): container finished" podID="b636e147-633a-403b-9858-2df91ba32080" containerID="77441092c5a25b5e615d1bc18c96ef154fbc5ee47b279c1d70d7262959aad73a" exitCode=0
Sep 10 16:30:04.320105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:04.320032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2d818c-29817630-7d9jn" event={"ID":"b636e147-633a-403b-9858-2df91ba32080","Type":"ContainerDied","Data":"77441092c5a25b5e615d1bc18c96ef154fbc5ee47b279c1d70d7262959aad73a"}
Sep 10 16:30:05.325401 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:05.325366    2580 generic.go:356] "Generic (PLEG): container finished" podID="55dc7d49-aa0a-4869-8bb6-6d6bb94d1314" containerID="b5fc6595162d1d10421b326f95d0ced460fe936f491b69b62bb6301743c3beb0" exitCode=0
Sep 10 16:30:05.325769 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:05.325453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817630-7nbjs" event={"ID":"55dc7d49-aa0a-4869-8bb6-6d6bb94d1314","Type":"ContainerDied","Data":"b5fc6595162d1d10421b326f95d0ced460fe936f491b69b62bb6301743c3beb0"}
Sep 10 16:30:05.475188 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:05.475163    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1abdaf-29817630-8jgcc"
Sep 10 16:30:05.505575 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:05.505551    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2d818c-29817630-7d9jn"
Sep 10 16:30:06.331473 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:06.331441    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1abdaf-29817630-8jgcc"
Sep 10 16:30:06.331919 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:06.331439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1abdaf-29817630-8jgcc" event={"ID":"be3a5461-0fcb-4eb1-a98a-bf5d3e1a84ab","Type":"ContainerDied","Data":"7288f75cf5f8911efbdcd9703e49cc6e22638cd4a4a4660b390a84810ea5d2e7"}
Sep 10 16:30:06.331919 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:06.331615    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7288f75cf5f8911efbdcd9703e49cc6e22638cd4a4a4660b390a84810ea5d2e7"
Sep 10 16:30:06.332833 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:06.332815    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2d818c-29817630-7d9jn"
Sep 10 16:30:06.332923 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:06.332845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2d818c-29817630-7d9jn" event={"ID":"b636e147-633a-403b-9858-2df91ba32080","Type":"ContainerDied","Data":"4b127b0c90e9cccb42898f2fefd4b068fbb91627c935525a642efd0a8d5390dd"}
Sep 10 16:30:06.332923 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:06.332883    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b127b0c90e9cccb42898f2fefd4b068fbb91627c935525a642efd0a8d5390dd"
Sep 10 16:30:06.466656 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:06.466633    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817630-7nbjs"
Sep 10 16:30:07.338763 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:07.338736    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817630-7nbjs"
Sep 10 16:30:07.339130 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:07.338738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817630-7nbjs" event={"ID":"55dc7d49-aa0a-4869-8bb6-6d6bb94d1314","Type":"ContainerDied","Data":"007eb4f98cb2fa498fe9ce7096d1cf4e84bcc7de0120d56fbf4dbc0360196541"}
Sep 10 16:30:07.339130 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:07.338857    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="007eb4f98cb2fa498fe9ce7096d1cf4e84bcc7de0120d56fbf4dbc0360196541"
Sep 10 16:30:15.918991 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:15.918959    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"]
Sep 10 16:30:15.919505 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:15.919203    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt" podUID="0b164e16-6a0b-48e7-b1ef-f095b923cc28" containerName="payload-pre-processing" containerID="cri-o://a6e46cf87eb4dd1de700f936f9efb95c8faefb900ffa6db47c6f8f9b875aae23" gracePeriod=30
Sep 10 16:30:16.172273 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.172222    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"
Sep 10 16:30:16.278042 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.278014    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b164e16-6a0b-48e7-b1ef-f095b923cc28-plugins-config-volume\") pod \"0b164e16-6a0b-48e7-b1ef-f095b923cc28\" (UID: \"0b164e16-6a0b-48e7-b1ef-f095b923cc28\") "
Sep 10 16:30:16.278213 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.278105    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl6h5\" (UniqueName: \"kubernetes.io/projected/0b164e16-6a0b-48e7-b1ef-f095b923cc28-kube-api-access-sl6h5\") pod \"0b164e16-6a0b-48e7-b1ef-f095b923cc28\" (UID: \"0b164e16-6a0b-48e7-b1ef-f095b923cc28\") "
Sep 10 16:30:16.278380 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.278353    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b164e16-6a0b-48e7-b1ef-f095b923cc28-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b164e16-6a0b-48e7-b1ef-f095b923cc28" (UID: "0b164e16-6a0b-48e7-b1ef-f095b923cc28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:30:16.280140 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.280109    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b164e16-6a0b-48e7-b1ef-f095b923cc28-kube-api-access-sl6h5" (OuterVolumeSpecName: "kube-api-access-sl6h5") pod "0b164e16-6a0b-48e7-b1ef-f095b923cc28" (UID: "0b164e16-6a0b-48e7-b1ef-f095b923cc28"). InnerVolumeSpecName "kube-api-access-sl6h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:30:16.378782 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.378743    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl6h5\" (UniqueName: \"kubernetes.io/projected/0b164e16-6a0b-48e7-b1ef-f095b923cc28-kube-api-access-sl6h5\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:30:16.378782 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.378779    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b164e16-6a0b-48e7-b1ef-f095b923cc28-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:30:16.384719 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.384688    2580 generic.go:356] "Generic (PLEG): container finished" podID="0b164e16-6a0b-48e7-b1ef-f095b923cc28" containerID="a6e46cf87eb4dd1de700f936f9efb95c8faefb900ffa6db47c6f8f9b875aae23" exitCode=0
Sep 10 16:30:16.384849 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.384733    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"
Sep 10 16:30:16.384849 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.384770    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt" event={"ID":"0b164e16-6a0b-48e7-b1ef-f095b923cc28","Type":"ContainerDied","Data":"a6e46cf87eb4dd1de700f936f9efb95c8faefb900ffa6db47c6f8f9b875aae23"}
Sep 10 16:30:16.384849 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.384840    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt" event={"ID":"0b164e16-6a0b-48e7-b1ef-f095b923cc28","Type":"ContainerDied","Data":"8f619f7d5b759bbb621b4b8abfe2f139b4c16c2405968ecdd15f3afea7dae20e"}
Sep 10 16:30:16.384983 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.384863    2580 scope.go:117] "RemoveContainer" containerID="a6e46cf87eb4dd1de700f936f9efb95c8faefb900ffa6db47c6f8f9b875aae23"
Sep 10 16:30:16.393680 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.393607    2580 scope.go:117] "RemoveContainer" containerID="a6e46cf87eb4dd1de700f936f9efb95c8faefb900ffa6db47c6f8f9b875aae23"
Sep 10 16:30:16.393875 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:30:16.393856    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e46cf87eb4dd1de700f936f9efb95c8faefb900ffa6db47c6f8f9b875aae23\": container with ID starting with a6e46cf87eb4dd1de700f936f9efb95c8faefb900ffa6db47c6f8f9b875aae23 not found: ID does not exist" containerID="a6e46cf87eb4dd1de700f936f9efb95c8faefb900ffa6db47c6f8f9b875aae23"
Sep 10 16:30:16.393919 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.393885    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e46cf87eb4dd1de700f936f9efb95c8faefb900ffa6db47c6f8f9b875aae23"} err="failed to get container status \"a6e46cf87eb4dd1de700f936f9efb95c8faefb900ffa6db47c6f8f9b875aae23\": rpc error: code = NotFound desc = could not find container \"a6e46cf87eb4dd1de700f936f9efb95c8faefb900ffa6db47c6f8f9b875aae23\": container with ID starting with a6e46cf87eb4dd1de700f936f9efb95c8faefb900ffa6db47c6f8f9b875aae23 not found: ID does not exist"
Sep 10 16:30:16.406957 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.406937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"]
Sep 10 16:30:16.410982 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:16.410961    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25d77f0b-764b86bf98-qrplt"]
Sep 10 16:30:18.254950 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:30:18.254912    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b164e16-6a0b-48e7-b1ef-f095b923cc28" path="/var/lib/kubelet/pods/0b164e16-6a0b-48e7-b1ef-f095b923cc28/volumes"
Sep 10 16:31:06.734300 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.734269    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"]
Sep 10 16:31:06.734739 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.734602    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b164e16-6a0b-48e7-b1ef-f095b923cc28" containerName="payload-pre-processing"
Sep 10 16:31:06.734739 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.734614    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b164e16-6a0b-48e7-b1ef-f095b923cc28" containerName="payload-pre-processing"
Sep 10 16:31:06.734739 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.734621    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be3a5461-0fcb-4eb1-a98a-bf5d3e1a84ab" containerName="cleanup"
Sep 10 16:31:06.734739 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.734627    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="be3a5461-0fcb-4eb1-a98a-bf5d3e1a84ab" containerName="cleanup"
Sep 10 16:31:06.734739 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.734639    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55dc7d49-aa0a-4869-8bb6-6d6bb94d1314" containerName="cleanup"
Sep 10 16:31:06.734739 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.734644    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="55dc7d49-aa0a-4869-8bb6-6d6bb94d1314" containerName="cleanup"
Sep 10 16:31:06.734739 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.734659    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b636e147-633a-403b-9858-2df91ba32080" containerName="cleanup"
Sep 10 16:31:06.734739 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.734664    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b636e147-633a-403b-9858-2df91ba32080" containerName="cleanup"
Sep 10 16:31:06.734739 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.734712    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="be3a5461-0fcb-4eb1-a98a-bf5d3e1a84ab" containerName="cleanup"
Sep 10 16:31:06.734739 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.734718    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="55dc7d49-aa0a-4869-8bb6-6d6bb94d1314" containerName="cleanup"
Sep 10 16:31:06.734739 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.734726    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b636e147-633a-403b-9858-2df91ba32080" containerName="cleanup"
Sep 10 16:31:06.734739 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.734731    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b164e16-6a0b-48e7-b1ef-f095b923cc28" containerName="payload-pre-processing"
Sep 10 16:31:06.737591 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.737575    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.739585 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.739561    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-a05a36b8-kserve-self-signed-certs\""
Sep 10 16:31:06.746508 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.746490    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"]
Sep 10 16:31:06.893783 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.893754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-home\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.893783 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.893785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-tmp-dir\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.893966 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.893809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-dshm\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.893966 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.893893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzd75\" (UniqueName: \"kubernetes.io/projected/857edefd-e8c0-4edd-b05d-a95be64aae72-kube-api-access-nzd75\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.893966 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.893931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-model-cache\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.894071 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.893970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857edefd-e8c0-4edd-b05d-a95be64aae72-tls-certs\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.994516 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.994451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-home\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.994516 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.994481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-tmp-dir\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.994516 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.994505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-dshm\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.994788 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.994552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzd75\" (UniqueName: \"kubernetes.io/projected/857edefd-e8c0-4edd-b05d-a95be64aae72-kube-api-access-nzd75\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.994788 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.994585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-model-cache\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.994788 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.994625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857edefd-e8c0-4edd-b05d-a95be64aae72-tls-certs\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.994938 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.994912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-home\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.995016 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.994989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-tmp-dir\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.995069 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.995046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-model-cache\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.996823 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.996805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-dshm\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:06.996942 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:06.996926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857edefd-e8c0-4edd-b05d-a95be64aae72-tls-certs\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:07.002616 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:07.002596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzd75\" (UniqueName: \"kubernetes.io/projected/857edefd-e8c0-4edd-b05d-a95be64aae72-kube-api-access-nzd75\") pod \"e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") " pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:07.047274 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:07.047251    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:07.324571 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:07.323764    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"]
Sep 10 16:31:07.611464 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:07.611383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn" event={"ID":"857edefd-e8c0-4edd-b05d-a95be64aae72","Type":"ContainerStarted","Data":"d3f3905f2f72db98ebfa8b99b2bd14ff4373e1acb3fc82e1f15f1794bf8047e1"}
Sep 10 16:31:07.611464 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:07.611421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn" event={"ID":"857edefd-e8c0-4edd-b05d-a95be64aae72","Type":"ContainerStarted","Data":"f86f76292553f2a188923f54ff444ab020d091a4f45e944e5acc2727b710d8f0"}
Sep 10 16:31:07.629360 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:07.629315    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn" podStartSLOduration=1.629301221 podStartE2EDuration="1.629301221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:31:07.62912037 +0000 UTC m=+825.830162782" watchObservedRunningTime="2026-09-10 16:31:07.629301221 +0000 UTC m=+825.830343632"
Sep 10 16:31:13.661617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.661514    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"]
Sep 10 16:31:13.666052 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.666029    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.669157 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.669121    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2d818c\"/\"autores-model-2d818c-kserve-self-signed-certs\""
Sep 10 16:31:13.669157 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.669129    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2d818c\"/\"default-dockercfg-xz65g\""
Sep 10 16:31:13.669410 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.669129    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2d818c\"/\"kube-root-ca.crt\""
Sep 10 16:31:13.669410 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.669178    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2d818c\"/\"openshift-service-ca.crt\""
Sep 10 16:31:13.672222 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.672195    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"]
Sep 10 16:31:13.752730 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.752705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm82p\" (UniqueName: \"kubernetes.io/projected/2820bd96-d277-4422-b51a-c58d75682325-kube-api-access-pm82p\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.752840 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.752735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-home\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.752840 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.752776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-model-cache\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.752840 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.752822    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2820bd96-d277-4422-b51a-c58d75682325-tls-certs\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.753009 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.752881    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-tmp-dir\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.753009 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.752962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-dshm\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.853734 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.853703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-tmp-dir\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.853847 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.853752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-dshm\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.853847 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.853791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm82p\" (UniqueName: \"kubernetes.io/projected/2820bd96-d277-4422-b51a-c58d75682325-kube-api-access-pm82p\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.853847 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.853814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-home\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.854007 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.853854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-model-cache\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.854007 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.853876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2820bd96-d277-4422-b51a-c58d75682325-tls-certs\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.854157 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.854134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-tmp-dir\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.854207 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.854180    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-home\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.854241 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.854207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-model-cache\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.856038 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.856016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-dshm\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.856278 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.856264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2820bd96-d277-4422-b51a-c58d75682325-tls-certs\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.862189 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.862172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm82p\" (UniqueName: \"kubernetes.io/projected/2820bd96-d277-4422-b51a-c58d75682325-kube-api-access-pm82p\") pod \"autores-model-2d818c-kserve-645bc8fc9f-q6npl\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") " pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:13.976833 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:13.976771    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:14.255367 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:31:14.255341    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2820bd96_d277_4422_b51a_c58d75682325.slice/crio-e16b29df70ee58b2d6326173b68f874ec45fbbf68089475227c791e0e1295064 WatchSource:0}: Error finding container e16b29df70ee58b2d6326173b68f874ec45fbbf68089475227c791e0e1295064: Status 404 returned error can't find the container with id e16b29df70ee58b2d6326173b68f874ec45fbbf68089475227c791e0e1295064
Sep 10 16:31:14.256327 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:14.256284    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"]
Sep 10 16:31:14.646089 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:14.646053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl" event={"ID":"2820bd96-d277-4422-b51a-c58d75682325","Type":"ContainerStarted","Data":"5d597e37c31df1b9b982ec2add1cedd28e97553aadfd56e0a8104f0988710643"}
Sep 10 16:31:14.646089 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:14.646091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl" event={"ID":"2820bd96-d277-4422-b51a-c58d75682325","Type":"ContainerStarted","Data":"e16b29df70ee58b2d6326173b68f874ec45fbbf68089475227c791e0e1295064"}
Sep 10 16:31:14.662571 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:14.662513    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl" podStartSLOduration=1.662498497 podStartE2EDuration="1.662498497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:31:14.661777879 +0000 UTC m=+832.862820292" watchObservedRunningTime="2026-09-10 16:31:14.662498497 +0000 UTC m=+832.863540924"
Sep 10 16:31:17.047553 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:17.047500    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:17.048005 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:17.047578    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:17.060218 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:17.060197    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:17.670727 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:17.670699    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:23.977426 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:23.977388    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:23.977850 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:23.977456    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:23.990030 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:23.990005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:24.703251 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:24.703221    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:33.846672 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.846641    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"]
Sep 10 16:31:33.850047 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.850014    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.852451 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.852431    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1abdaf\"/\"default-dockercfg-fn6m8\""
Sep 10 16:31:33.853167 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.853143    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1abdaf\"/\"kube-root-ca.crt\""
Sep 10 16:31:33.853278 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.853143    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1abdaf\"/\"test-model-1abdaf-kserve-self-signed-certs\""
Sep 10 16:31:33.853278 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.853166    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1abdaf\"/\"openshift-service-ca.crt\""
Sep 10 16:31:33.867328 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.867304    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"]
Sep 10 16:31:33.896750 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.896728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3125dadf-c703-4ab0-aa03-322c6de2b827-tls-certs\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.896847 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.896768    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-tmp-dir\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.896847 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.896801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk5t7\" (UniqueName: \"kubernetes.io/projected/3125dadf-c703-4ab0-aa03-322c6de2b827-kube-api-access-wk5t7\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.896931 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.896871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-dshm\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.896931 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.896902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-home\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.897004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.896945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-model-cache\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.998283 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.998259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-dshm\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.998395 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.998289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-home\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.998395 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.998317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-model-cache\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.998395 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.998346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3125dadf-c703-4ab0-aa03-322c6de2b827-tls-certs\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.998561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.998451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-tmp-dir\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.998561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.998514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk5t7\" (UniqueName: \"kubernetes.io/projected/3125dadf-c703-4ab0-aa03-322c6de2b827-kube-api-access-wk5t7\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.998783 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.998755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-model-cache\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.998897 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.998818    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-home\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:33.998897 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:33.998855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-tmp-dir\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:34.000590 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:34.000569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-dshm\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:34.000839 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:34.000824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3125dadf-c703-4ab0-aa03-322c6de2b827-tls-certs\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:34.006584 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:34.006564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk5t7\" (UniqueName: \"kubernetes.io/projected/3125dadf-c703-4ab0-aa03-322c6de2b827-kube-api-access-wk5t7\") pod \"test-model-1abdaf-kserve-f754fb599-hwr9s\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") " pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:34.159058 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:34.159000    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:34.427897 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:34.427874    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"]
Sep 10 16:31:34.429618 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:31:34.429592    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3125dadf_c703_4ab0_aa03_322c6de2b827.slice/crio-43372ce72b808b92d258a989f71a5a165dbafb88e655b4edcb4d1a372a6458f7 WatchSource:0}: Error finding container 43372ce72b808b92d258a989f71a5a165dbafb88e655b4edcb4d1a372a6458f7: Status 404 returned error can't find the container with id 43372ce72b808b92d258a989f71a5a165dbafb88e655b4edcb4d1a372a6458f7
Sep 10 16:31:34.737572 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:34.737465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s" event={"ID":"3125dadf-c703-4ab0-aa03-322c6de2b827","Type":"ContainerStarted","Data":"7b5aeb825a43c3c973d81c3b8ff48babd5c496bbb09060eb55efdc5b8bb45d74"}
Sep 10 16:31:34.737572 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:34.737502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s" event={"ID":"3125dadf-c703-4ab0-aa03-322c6de2b827","Type":"ContainerStarted","Data":"43372ce72b808b92d258a989f71a5a165dbafb88e655b4edcb4d1a372a6458f7"}
Sep 10 16:31:34.756851 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:34.756804    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s" podStartSLOduration=1.7567899059999998 podStartE2EDuration="1.756789906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:31:34.755662959 +0000 UTC m=+852.956705382" watchObservedRunningTime="2026-09-10 16:31:34.756789906 +0000 UTC m=+852.957832316"
Sep 10 16:31:36.281429 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:36.281388    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"]
Sep 10 16:31:37.072750 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:37.072718    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"]
Sep 10 16:31:44.159793 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:44.159758    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:44.160272 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:44.159841    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:44.172308 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:44.172289    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:44.794519 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:44.794489    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:31:47.990520 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:47.990490    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2d818c-29817630-7d9jn"]
Sep 10 16:31:47.997338 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:47.997312    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2d818c-29817630-7d9jn"]
Sep 10 16:31:48.020166 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.020131    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"]
Sep 10 16:31:48.020394 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.020367    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v" podUID="5bc71372-93fa-440a-bd6e-af84554d02a1" containerName="payload-processing" containerID="cri-o://37b559913c10724c89bf578da7ab1b44033e9006768825a20c45733de3e2a654" gracePeriod=30
Sep 10 16:31:48.171961 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.171921    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v" podUID="5bc71372-93fa-440a-bd6e-af84554d02a1" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.29:9004: connect: connection refused"
Sep 10 16:31:48.254678 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.254652    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b636e147-633a-403b-9858-2df91ba32080" path="/var/lib/kubelet/pods/b636e147-633a-403b-9858-2df91ba32080/volumes"
Sep 10 16:31:48.273643 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.273621    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"
Sep 10 16:31:48.317855 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.317819    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc71372-93fa-440a-bd6e-af84554d02a1-plugins-config-volume\") pod \"5bc71372-93fa-440a-bd6e-af84554d02a1\" (UID: \"5bc71372-93fa-440a-bd6e-af84554d02a1\") "
Sep 10 16:31:48.317998 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.317932    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkffx\" (UniqueName: \"kubernetes.io/projected/5bc71372-93fa-440a-bd6e-af84554d02a1-kube-api-access-mkffx\") pod \"5bc71372-93fa-440a-bd6e-af84554d02a1\" (UID: \"5bc71372-93fa-440a-bd6e-af84554d02a1\") "
Sep 10 16:31:48.318331 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.318305    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc71372-93fa-440a-bd6e-af84554d02a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5bc71372-93fa-440a-bd6e-af84554d02a1" (UID: "5bc71372-93fa-440a-bd6e-af84554d02a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:31:48.320032 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.320007    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc71372-93fa-440a-bd6e-af84554d02a1-kube-api-access-mkffx" (OuterVolumeSpecName: "kube-api-access-mkffx") pod "5bc71372-93fa-440a-bd6e-af84554d02a1" (UID: "5bc71372-93fa-440a-bd6e-af84554d02a1"). InnerVolumeSpecName "kube-api-access-mkffx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:31:48.418891 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.418862    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc71372-93fa-440a-bd6e-af84554d02a1-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:31:48.418891 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.418887    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkffx\" (UniqueName: \"kubernetes.io/projected/5bc71372-93fa-440a-bd6e-af84554d02a1-kube-api-access-mkffx\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:31:48.802492 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.802462    2580 generic.go:356] "Generic (PLEG): container finished" podID="5bc71372-93fa-440a-bd6e-af84554d02a1" containerID="37b559913c10724c89bf578da7ab1b44033e9006768825a20c45733de3e2a654" exitCode=0
Sep 10 16:31:48.802665 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.802505    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"
Sep 10 16:31:48.802665 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.802560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v" event={"ID":"5bc71372-93fa-440a-bd6e-af84554d02a1","Type":"ContainerDied","Data":"37b559913c10724c89bf578da7ab1b44033e9006768825a20c45733de3e2a654"}
Sep 10 16:31:48.802665 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.802610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v" event={"ID":"5bc71372-93fa-440a-bd6e-af84554d02a1","Type":"ContainerDied","Data":"fd35dab4ffdf9793be5b6e4ae4ef356643a56b0bab17dc7189f68c407c552d00"}
Sep 10 16:31:48.802665 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.802635    2580 scope.go:117] "RemoveContainer" containerID="37b559913c10724c89bf578da7ab1b44033e9006768825a20c45733de3e2a654"
Sep 10 16:31:48.811269 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.811253    2580 scope.go:117] "RemoveContainer" containerID="37b559913c10724c89bf578da7ab1b44033e9006768825a20c45733de3e2a654"
Sep 10 16:31:48.811565 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:31:48.811546    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b559913c10724c89bf578da7ab1b44033e9006768825a20c45733de3e2a654\": container with ID starting with 37b559913c10724c89bf578da7ab1b44033e9006768825a20c45733de3e2a654 not found: ID does not exist" containerID="37b559913c10724c89bf578da7ab1b44033e9006768825a20c45733de3e2a654"
Sep 10 16:31:48.811640 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.811571    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b559913c10724c89bf578da7ab1b44033e9006768825a20c45733de3e2a654"} err="failed to get container status \"37b559913c10724c89bf578da7ab1b44033e9006768825a20c45733de3e2a654\": rpc error: code = NotFound desc = could not find container \"37b559913c10724c89bf578da7ab1b44033e9006768825a20c45733de3e2a654\": container with ID starting with 37b559913c10724c89bf578da7ab1b44033e9006768825a20c45733de3e2a654 not found: ID does not exist"
Sep 10 16:31:48.823025 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.823004    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"]
Sep 10 16:31:48.826383 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:48.826363    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2d818c-7b955d8589-hql4v"]
Sep 10 16:31:50.258865 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:50.258826    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bc71372-93fa-440a-bd6e-af84554d02a1" path="/var/lib/kubelet/pods/5bc71372-93fa-440a-bd6e-af84554d02a1/volumes"
Sep 10 16:31:51.300838 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.300770    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn" podUID="857edefd-e8c0-4edd-b05d-a95be64aae72" containerName="main" containerID="cri-o://d3f3905f2f72db98ebfa8b99b2bd14ff4373e1acb3fc82e1f15f1794bf8047e1" gracePeriod=45
Sep 10 16:31:51.549572 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.549549    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:51.646396 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.646327    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857edefd-e8c0-4edd-b05d-a95be64aae72-tls-certs\") pod \"857edefd-e8c0-4edd-b05d-a95be64aae72\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") "
Sep 10 16:31:51.646396 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.646363    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-tmp-dir\") pod \"857edefd-e8c0-4edd-b05d-a95be64aae72\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") "
Sep 10 16:31:51.646631 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.646403    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-dshm\") pod \"857edefd-e8c0-4edd-b05d-a95be64aae72\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") "
Sep 10 16:31:51.646631 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.646424    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-model-cache\") pod \"857edefd-e8c0-4edd-b05d-a95be64aae72\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") "
Sep 10 16:31:51.646631 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.646477    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzd75\" (UniqueName: \"kubernetes.io/projected/857edefd-e8c0-4edd-b05d-a95be64aae72-kube-api-access-nzd75\") pod \"857edefd-e8c0-4edd-b05d-a95be64aae72\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") "
Sep 10 16:31:51.646631 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.646513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-home\") pod \"857edefd-e8c0-4edd-b05d-a95be64aae72\" (UID: \"857edefd-e8c0-4edd-b05d-a95be64aae72\") "
Sep 10 16:31:51.646820 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.646713    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "857edefd-e8c0-4edd-b05d-a95be64aae72" (UID: "857edefd-e8c0-4edd-b05d-a95be64aae72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:31:51.646820 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.646754    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-model-cache" (OuterVolumeSpecName: "model-cache") pod "857edefd-e8c0-4edd-b05d-a95be64aae72" (UID: "857edefd-e8c0-4edd-b05d-a95be64aae72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:31:51.646915 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.646892    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-home" (OuterVolumeSpecName: "home") pod "857edefd-e8c0-4edd-b05d-a95be64aae72" (UID: "857edefd-e8c0-4edd-b05d-a95be64aae72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:31:51.648598 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.648568    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/857edefd-e8c0-4edd-b05d-a95be64aae72-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "857edefd-e8c0-4edd-b05d-a95be64aae72" (UID: "857edefd-e8c0-4edd-b05d-a95be64aae72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:31:51.648598 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.648588    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-dshm" (OuterVolumeSpecName: "dshm") pod "857edefd-e8c0-4edd-b05d-a95be64aae72" (UID: "857edefd-e8c0-4edd-b05d-a95be64aae72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:31:51.648732 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.648612    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857edefd-e8c0-4edd-b05d-a95be64aae72-kube-api-access-nzd75" (OuterVolumeSpecName: "kube-api-access-nzd75") pod "857edefd-e8c0-4edd-b05d-a95be64aae72" (UID: "857edefd-e8c0-4edd-b05d-a95be64aae72"). InnerVolumeSpecName "kube-api-access-nzd75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:31:51.747881 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.747856    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:31:51.747881 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.747878    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:31:51.748022 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.747891    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:31:51.748022 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.747905    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzd75\" (UniqueName: \"kubernetes.io/projected/857edefd-e8c0-4edd-b05d-a95be64aae72-kube-api-access-nzd75\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:31:51.748022 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.747916    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857edefd-e8c0-4edd-b05d-a95be64aae72-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:31:51.748022 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.747928    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857edefd-e8c0-4edd-b05d-a95be64aae72-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:31:51.818719 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.818694    2580 generic.go:356] "Generic (PLEG): container finished" podID="857edefd-e8c0-4edd-b05d-a95be64aae72" containerID="d3f3905f2f72db98ebfa8b99b2bd14ff4373e1acb3fc82e1f15f1794bf8047e1" exitCode=0
Sep 10 16:31:51.818827 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.818736    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"
Sep 10 16:31:51.818827 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.818782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn" event={"ID":"857edefd-e8c0-4edd-b05d-a95be64aae72","Type":"ContainerDied","Data":"d3f3905f2f72db98ebfa8b99b2bd14ff4373e1acb3fc82e1f15f1794bf8047e1"}
Sep 10 16:31:51.818827 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.818821    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn" event={"ID":"857edefd-e8c0-4edd-b05d-a95be64aae72","Type":"ContainerDied","Data":"f86f76292553f2a188923f54ff444ab020d091a4f45e944e5acc2727b710d8f0"}
Sep 10 16:31:51.818953 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.818838    2580 scope.go:117] "RemoveContainer" containerID="d3f3905f2f72db98ebfa8b99b2bd14ff4373e1acb3fc82e1f15f1794bf8047e1"
Sep 10 16:31:51.827701 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.827675    2580 scope.go:117] "RemoveContainer" containerID="d3f3905f2f72db98ebfa8b99b2bd14ff4373e1acb3fc82e1f15f1794bf8047e1"
Sep 10 16:31:51.827974 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:31:51.827954    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f3905f2f72db98ebfa8b99b2bd14ff4373e1acb3fc82e1f15f1794bf8047e1\": container with ID starting with d3f3905f2f72db98ebfa8b99b2bd14ff4373e1acb3fc82e1f15f1794bf8047e1 not found: ID does not exist" containerID="d3f3905f2f72db98ebfa8b99b2bd14ff4373e1acb3fc82e1f15f1794bf8047e1"
Sep 10 16:31:51.828038 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.827985    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f3905f2f72db98ebfa8b99b2bd14ff4373e1acb3fc82e1f15f1794bf8047e1"} err="failed to get container status \"d3f3905f2f72db98ebfa8b99b2bd14ff4373e1acb3fc82e1f15f1794bf8047e1\": rpc error: code = NotFound desc = could not find container \"d3f3905f2f72db98ebfa8b99b2bd14ff4373e1acb3fc82e1f15f1794bf8047e1\": container with ID starting with d3f3905f2f72db98ebfa8b99b2bd14ff4373e1acb3fc82e1f15f1794bf8047e1 not found: ID does not exist"
Sep 10 16:31:51.841675 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.841654    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"]
Sep 10 16:31:51.849685 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:51.849667    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-a05a36b8-kserve-75b787fc75-89jfn"]
Sep 10 16:31:52.095936 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.095903    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl" podUID="2820bd96-d277-4422-b51a-c58d75682325" containerName="main" containerID="cri-o://5d597e37c31df1b9b982ec2add1cedd28e97553aadfd56e0a8104f0988710643" gracePeriod=45
Sep 10 16:31:52.256126 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.256087    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="857edefd-e8c0-4edd-b05d-a95be64aae72" path="/var/lib/kubelet/pods/857edefd-e8c0-4edd-b05d-a95be64aae72/volumes"
Sep 10 16:31:52.349129 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.349082    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:52.453401 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.453374    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-tmp-dir\") pod \"2820bd96-d277-4422-b51a-c58d75682325\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") "
Sep 10 16:31:52.453556 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.453420    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2820bd96-d277-4422-b51a-c58d75682325-tls-certs\") pod \"2820bd96-d277-4422-b51a-c58d75682325\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") "
Sep 10 16:31:52.453556 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.453448    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-model-cache\") pod \"2820bd96-d277-4422-b51a-c58d75682325\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") "
Sep 10 16:31:52.453556 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.453465    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-dshm\") pod \"2820bd96-d277-4422-b51a-c58d75682325\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") "
Sep 10 16:31:52.453556 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.453493    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm82p\" (UniqueName: \"kubernetes.io/projected/2820bd96-d277-4422-b51a-c58d75682325-kube-api-access-pm82p\") pod \"2820bd96-d277-4422-b51a-c58d75682325\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") "
Sep 10 16:31:52.453556 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.453525    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-home\") pod \"2820bd96-d277-4422-b51a-c58d75682325\" (UID: \"2820bd96-d277-4422-b51a-c58d75682325\") "
Sep 10 16:31:52.453820 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.453717    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2820bd96-d277-4422-b51a-c58d75682325" (UID: "2820bd96-d277-4422-b51a-c58d75682325"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:31:52.453820 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.453751    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-model-cache" (OuterVolumeSpecName: "model-cache") pod "2820bd96-d277-4422-b51a-c58d75682325" (UID: "2820bd96-d277-4422-b51a-c58d75682325"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:31:52.453935 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.453826    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:31:52.453935 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.453844    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:31:52.453935 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.453918    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-home" (OuterVolumeSpecName: "home") pod "2820bd96-d277-4422-b51a-c58d75682325" (UID: "2820bd96-d277-4422-b51a-c58d75682325"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:31:52.455746 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.455724    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-dshm" (OuterVolumeSpecName: "dshm") pod "2820bd96-d277-4422-b51a-c58d75682325" (UID: "2820bd96-d277-4422-b51a-c58d75682325"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:31:52.455853 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.455781    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2820bd96-d277-4422-b51a-c58d75682325-kube-api-access-pm82p" (OuterVolumeSpecName: "kube-api-access-pm82p") pod "2820bd96-d277-4422-b51a-c58d75682325" (UID: "2820bd96-d277-4422-b51a-c58d75682325"). InnerVolumeSpecName "kube-api-access-pm82p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:31:52.455896 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.455869    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2820bd96-d277-4422-b51a-c58d75682325-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2820bd96-d277-4422-b51a-c58d75682325" (UID: "2820bd96-d277-4422-b51a-c58d75682325"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:31:52.554915 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.554889    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2820bd96-d277-4422-b51a-c58d75682325-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:31:52.554915 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.554912    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:31:52.554915 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.554922    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm82p\" (UniqueName: \"kubernetes.io/projected/2820bd96-d277-4422-b51a-c58d75682325-kube-api-access-pm82p\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:31:52.555094 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.554929    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2820bd96-d277-4422-b51a-c58d75682325-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:31:52.824928 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.824893    2580 generic.go:356] "Generic (PLEG): container finished" podID="2820bd96-d277-4422-b51a-c58d75682325" containerID="5d597e37c31df1b9b982ec2add1cedd28e97553aadfd56e0a8104f0988710643" exitCode=0
Sep 10 16:31:52.825115 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.824936    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"
Sep 10 16:31:52.825115 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.824996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl" event={"ID":"2820bd96-d277-4422-b51a-c58d75682325","Type":"ContainerDied","Data":"5d597e37c31df1b9b982ec2add1cedd28e97553aadfd56e0a8104f0988710643"}
Sep 10 16:31:52.825115 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.825061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl" event={"ID":"2820bd96-d277-4422-b51a-c58d75682325","Type":"ContainerDied","Data":"e16b29df70ee58b2d6326173b68f874ec45fbbf68089475227c791e0e1295064"}
Sep 10 16:31:52.825115 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.825094    2580 scope.go:117] "RemoveContainer" containerID="5d597e37c31df1b9b982ec2add1cedd28e97553aadfd56e0a8104f0988710643"
Sep 10 16:31:52.834744 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.834726    2580 scope.go:117] "RemoveContainer" containerID="5d597e37c31df1b9b982ec2add1cedd28e97553aadfd56e0a8104f0988710643"
Sep 10 16:31:52.835008 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:31:52.834991    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d597e37c31df1b9b982ec2add1cedd28e97553aadfd56e0a8104f0988710643\": container with ID starting with 5d597e37c31df1b9b982ec2add1cedd28e97553aadfd56e0a8104f0988710643 not found: ID does not exist" containerID="5d597e37c31df1b9b982ec2add1cedd28e97553aadfd56e0a8104f0988710643"
Sep 10 16:31:52.835067 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.835016    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d597e37c31df1b9b982ec2add1cedd28e97553aadfd56e0a8104f0988710643"} err="failed to get container status \"5d597e37c31df1b9b982ec2add1cedd28e97553aadfd56e0a8104f0988710643\": rpc error: code = NotFound desc = could not find container \"5d597e37c31df1b9b982ec2add1cedd28e97553aadfd56e0a8104f0988710643\": container with ID starting with 5d597e37c31df1b9b982ec2add1cedd28e97553aadfd56e0a8104f0988710643 not found: ID does not exist"
Sep 10 16:31:52.848109 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.848088    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"]
Sep 10 16:31:52.856429 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:52.856408    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-2d818c/autores-model-2d818c-kserve-645bc8fc9f-q6npl"]
Sep 10 16:31:54.254387 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:31:54.254356    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2820bd96-d277-4422-b51a-c58d75682325" path="/var/lib/kubelet/pods/2820bd96-d277-4422-b51a-c58d75682325/volumes"
Sep 10 16:32:03.713299 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.713264    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"]
Sep 10 16:32:03.713696 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.713681    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2820bd96-d277-4422-b51a-c58d75682325" containerName="main"
Sep 10 16:32:03.713744 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.713698    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2820bd96-d277-4422-b51a-c58d75682325" containerName="main"
Sep 10 16:32:03.713744 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.713710    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bc71372-93fa-440a-bd6e-af84554d02a1" containerName="payload-processing"
Sep 10 16:32:03.713744 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.713716    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bc71372-93fa-440a-bd6e-af84554d02a1" containerName="payload-processing"
Sep 10 16:32:03.713744 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.713729    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="857edefd-e8c0-4edd-b05d-a95be64aae72" containerName="main"
Sep 10 16:32:03.713744 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.713735    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="857edefd-e8c0-4edd-b05d-a95be64aae72" containerName="main"
Sep 10 16:32:03.713890 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.713788    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2820bd96-d277-4422-b51a-c58d75682325" containerName="main"
Sep 10 16:32:03.713890 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.713798    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="857edefd-e8c0-4edd-b05d-a95be64aae72" containerName="main"
Sep 10 16:32:03.713890 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.713805    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bc71372-93fa-440a-bd6e-af84554d02a1" containerName="payload-processing"
Sep 10 16:32:03.718094 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.718075    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"
Sep 10 16:32:03.720383 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.720350    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-09da25\""
Sep 10 16:32:03.720487 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.720471    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-09da25-dockercfg-wbxk4\""
Sep 10 16:32:03.726114 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.726093    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"]
Sep 10 16:32:03.848905 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.848879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a93ec43-e931-4299-92ab-b092539de479-plugins-config-volume\") pod \"payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9\" (UID: \"4a93ec43-e931-4299-92ab-b092539de479\") " pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"
Sep 10 16:32:03.849038 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.848919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x9k5\" (UniqueName: \"kubernetes.io/projected/4a93ec43-e931-4299-92ab-b092539de479-kube-api-access-2x9k5\") pod \"payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9\" (UID: \"4a93ec43-e931-4299-92ab-b092539de479\") " pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"
Sep 10 16:32:03.950126 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.950098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a93ec43-e931-4299-92ab-b092539de479-plugins-config-volume\") pod \"payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9\" (UID: \"4a93ec43-e931-4299-92ab-b092539de479\") " pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"
Sep 10 16:32:03.950238 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.950134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x9k5\" (UniqueName: \"kubernetes.io/projected/4a93ec43-e931-4299-92ab-b092539de479-kube-api-access-2x9k5\") pod \"payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9\" (UID: \"4a93ec43-e931-4299-92ab-b092539de479\") " pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"
Sep 10 16:32:03.950813 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.950795    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a93ec43-e931-4299-92ab-b092539de479-plugins-config-volume\") pod \"payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9\" (UID: \"4a93ec43-e931-4299-92ab-b092539de479\") " pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"
Sep 10 16:32:03.959360 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:03.959337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x9k5\" (UniqueName: \"kubernetes.io/projected/4a93ec43-e931-4299-92ab-b092539de479-kube-api-access-2x9k5\") pod \"payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9\" (UID: \"4a93ec43-e931-4299-92ab-b092539de479\") " pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"
Sep 10 16:32:04.028965 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:04.028943    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"
Sep 10 16:32:04.300053 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:04.300027    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"]
Sep 10 16:32:04.302034 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:32:04.302003    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a93ec43_e931_4299_92ab_b092539de479.slice/crio-ac057bfcda2be0312bf600970e6f2a0747f132b958788141e0010c11cc58c182 WatchSource:0}: Error finding container ac057bfcda2be0312bf600970e6f2a0747f132b958788141e0010c11cc58c182: Status 404 returned error can't find the container with id ac057bfcda2be0312bf600970e6f2a0747f132b958788141e0010c11cc58c182
Sep 10 16:32:04.881431 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:04.881392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9" event={"ID":"4a93ec43-e931-4299-92ab-b092539de479","Type":"ContainerStarted","Data":"6bee92f3221d153b864dcc4b6cde3e9370732f0d99bc55cfe6e7cb3930334361"}
Sep 10 16:32:04.881840 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:04.881456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9" event={"ID":"4a93ec43-e931-4299-92ab-b092539de479","Type":"ContainerStarted","Data":"ac057bfcda2be0312bf600970e6f2a0747f132b958788141e0010c11cc58c182"}
Sep 10 16:32:04.881840 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:04.881486    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"
Sep 10 16:32:04.898717 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:04.898649    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9" podStartSLOduration=1.898631288 podStartE2EDuration="1.898631288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:32:04.898041935 +0000 UTC m=+883.099084346" watchObservedRunningTime="2026-09-10 16:32:04.898631288 +0000 UTC m=+883.099673701"
Sep 10 16:32:05.934148 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:05.934114    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"]
Sep 10 16:32:15.889466 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:15.889390    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"
Sep 10 16:32:20.953404 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:20.953339    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg" podUID="ad232a58-d8e1-4a88-88b6-264dbd967238" containerName="main" containerID="cri-o://1fe8a2f256488006b18f03ecb515b7225f34dc83e79512247e829d943b2931f4" gracePeriod=45
Sep 10 16:32:21.199680 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.199655    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:32:21.390158 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.390125    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msmpg\" (UniqueName: \"kubernetes.io/projected/ad232a58-d8e1-4a88-88b6-264dbd967238-kube-api-access-msmpg\") pod \"ad232a58-d8e1-4a88-88b6-264dbd967238\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") "
Sep 10 16:32:21.390329 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.390198    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-tmp-dir\") pod \"ad232a58-d8e1-4a88-88b6-264dbd967238\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") "
Sep 10 16:32:21.390329 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.390222    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-model-cache\") pod \"ad232a58-d8e1-4a88-88b6-264dbd967238\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") "
Sep 10 16:32:21.390329 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.390291    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-dshm\") pod \"ad232a58-d8e1-4a88-88b6-264dbd967238\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") "
Sep 10 16:32:21.390329 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.390316    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-home\") pod \"ad232a58-d8e1-4a88-88b6-264dbd967238\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") "
Sep 10 16:32:21.390486 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.390359    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad232a58-d8e1-4a88-88b6-264dbd967238-tls-certs\") pod \"ad232a58-d8e1-4a88-88b6-264dbd967238\" (UID: \"ad232a58-d8e1-4a88-88b6-264dbd967238\") "
Sep 10 16:32:21.390571 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.390477    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad232a58-d8e1-4a88-88b6-264dbd967238" (UID: "ad232a58-d8e1-4a88-88b6-264dbd967238"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:32:21.390571 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.390492    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad232a58-d8e1-4a88-88b6-264dbd967238" (UID: "ad232a58-d8e1-4a88-88b6-264dbd967238"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:32:21.390684 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.390649    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:32:21.390684 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.390662    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:32:21.390935 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.390902    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-home" (OuterVolumeSpecName: "home") pod "ad232a58-d8e1-4a88-88b6-264dbd967238" (UID: "ad232a58-d8e1-4a88-88b6-264dbd967238"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:32:21.392388 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.392362    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-dshm" (OuterVolumeSpecName: "dshm") pod "ad232a58-d8e1-4a88-88b6-264dbd967238" (UID: "ad232a58-d8e1-4a88-88b6-264dbd967238"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:32:21.392481 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.392386    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad232a58-d8e1-4a88-88b6-264dbd967238-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad232a58-d8e1-4a88-88b6-264dbd967238" (UID: "ad232a58-d8e1-4a88-88b6-264dbd967238"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:32:21.392481 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.392409    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad232a58-d8e1-4a88-88b6-264dbd967238-kube-api-access-msmpg" (OuterVolumeSpecName: "kube-api-access-msmpg") pod "ad232a58-d8e1-4a88-88b6-264dbd967238" (UID: "ad232a58-d8e1-4a88-88b6-264dbd967238"). InnerVolumeSpecName "kube-api-access-msmpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:32:21.491714 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.491682    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:32:21.491714 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.491711    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad232a58-d8e1-4a88-88b6-264dbd967238-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:32:21.491854 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.491719    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad232a58-d8e1-4a88-88b6-264dbd967238-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:32:21.491854 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.491730    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msmpg\" (UniqueName: \"kubernetes.io/projected/ad232a58-d8e1-4a88-88b6-264dbd967238-kube-api-access-msmpg\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:32:21.960201 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.960164    2580 generic.go:356] "Generic (PLEG): container finished" podID="ad232a58-d8e1-4a88-88b6-264dbd967238" containerID="1fe8a2f256488006b18f03ecb515b7225f34dc83e79512247e829d943b2931f4" exitCode=0
Sep 10 16:32:21.960617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.960208    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"
Sep 10 16:32:21.960617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.960254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg" event={"ID":"ad232a58-d8e1-4a88-88b6-264dbd967238","Type":"ContainerDied","Data":"1fe8a2f256488006b18f03ecb515b7225f34dc83e79512247e829d943b2931f4"}
Sep 10 16:32:21.960617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.960300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg" event={"ID":"ad232a58-d8e1-4a88-88b6-264dbd967238","Type":"ContainerDied","Data":"819252658c7862b62ff57c8236e56216b0ab3a1f6f47d0adf163885436c6f918"}
Sep 10 16:32:21.960617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.960316    2580 scope.go:117] "RemoveContainer" containerID="1fe8a2f256488006b18f03ecb515b7225f34dc83e79512247e829d943b2931f4"
Sep 10 16:32:21.969135 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.969119    2580 scope.go:117] "RemoveContainer" containerID="1fe8a2f256488006b18f03ecb515b7225f34dc83e79512247e829d943b2931f4"
Sep 10 16:32:21.969400 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:32:21.969381    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe8a2f256488006b18f03ecb515b7225f34dc83e79512247e829d943b2931f4\": container with ID starting with 1fe8a2f256488006b18f03ecb515b7225f34dc83e79512247e829d943b2931f4 not found: ID does not exist" containerID="1fe8a2f256488006b18f03ecb515b7225f34dc83e79512247e829d943b2931f4"
Sep 10 16:32:21.969441 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.969409    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe8a2f256488006b18f03ecb515b7225f34dc83e79512247e829d943b2931f4"} err="failed to get container status \"1fe8a2f256488006b18f03ecb515b7225f34dc83e79512247e829d943b2931f4\": rpc error: code = NotFound desc = could not find container \"1fe8a2f256488006b18f03ecb515b7225f34dc83e79512247e829d943b2931f4\": container with ID starting with 1fe8a2f256488006b18f03ecb515b7225f34dc83e79512247e829d943b2931f4 not found: ID does not exist"
Sep 10 16:32:21.980656 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.980637    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"]
Sep 10 16:32:21.983516 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:21.983494    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-a05a36b8-kserve-75cd9c6b5-lhnkg"]
Sep 10 16:32:22.255255 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:22.255182    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad232a58-d8e1-4a88-88b6-264dbd967238" path="/var/lib/kubelet/pods/ad232a58-d8e1-4a88-88b6-264dbd967238/volumes"
Sep 10 16:32:23.538472 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.538440    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"]
Sep 10 16:32:23.538956 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.538940    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad232a58-d8e1-4a88-88b6-264dbd967238" containerName="main"
Sep 10 16:32:23.538999 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.538960    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad232a58-d8e1-4a88-88b6-264dbd967238" containerName="main"
Sep 10 16:32:23.539072 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.539062    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad232a58-d8e1-4a88-88b6-264dbd967238" containerName="main"
Sep 10 16:32:23.543679 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.543658    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.546620 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.546587    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9f6082\"/\"default-dockercfg-shwrl\""
Sep 10 16:32:23.546773 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.546736    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9f6082\"/\"openshift-service-ca.crt\""
Sep 10 16:32:23.546773 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.546745    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9f6082\"/\"kube-root-ca.crt\""
Sep 10 16:32:23.547035 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.547005    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9f6082\"/\"test-model-9f6082-kserve-self-signed-certs\""
Sep 10 16:32:23.548957 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.548936    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"]
Sep 10 16:32:23.609322 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.609289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-dshm\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.609479 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.609331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-model-cache\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.609479 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.609353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616050b2-189e-4b56-a049-314f73837e68-tls-certs\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.609479 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.609422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg746\" (UniqueName: \"kubernetes.io/projected/616050b2-189e-4b56-a049-314f73837e68-kube-api-access-tg746\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.609608 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.609495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-home\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.609608 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.609568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-tmp-dir\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.710431 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.710396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-home\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.710617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.710439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-tmp-dir\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.710617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.710484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-dshm\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.710617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.710522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-model-cache\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.710617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.710571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616050b2-189e-4b56-a049-314f73837e68-tls-certs\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.710815 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.710618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg746\" (UniqueName: \"kubernetes.io/projected/616050b2-189e-4b56-a049-314f73837e68-kube-api-access-tg746\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.710875 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.710844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-home\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.710931 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.710907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-tmp-dir\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.710998 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.710975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-model-cache\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.712867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.712845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-dshm\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.713032 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.713014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616050b2-189e-4b56-a049-314f73837e68-tls-certs\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.717690 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.717672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg746\" (UniqueName: \"kubernetes.io/projected/616050b2-189e-4b56-a049-314f73837e68-kube-api-access-tg746\") pod \"test-model-9f6082-kserve-79c74867c9-pr7vg\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") " pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:23.854056 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:23.853978    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:24.128060 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:24.128037    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"]
Sep 10 16:32:24.129887 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:32:24.129861    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod616050b2_189e_4b56_a049_314f73837e68.slice/crio-65bcc42bc067799b53dc9863d010a4e24eb9527c22fc6803b8d04f54b103b0cf WatchSource:0}: Error finding container 65bcc42bc067799b53dc9863d010a4e24eb9527c22fc6803b8d04f54b103b0cf: Status 404 returned error can't find the container with id 65bcc42bc067799b53dc9863d010a4e24eb9527c22fc6803b8d04f54b103b0cf
Sep 10 16:32:24.975265 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:24.975231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg" event={"ID":"616050b2-189e-4b56-a049-314f73837e68","Type":"ContainerStarted","Data":"fb1a24e25083c43ad8e5413b4bc4f59bec595670592f1999dfccbb5f3e90b9d1"}
Sep 10 16:32:24.975652 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:24.975271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg" event={"ID":"616050b2-189e-4b56-a049-314f73837e68","Type":"ContainerStarted","Data":"65bcc42bc067799b53dc9863d010a4e24eb9527c22fc6803b8d04f54b103b0cf"}
Sep 10 16:32:24.993164 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:24.993119    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg" podStartSLOduration=1.9931049220000001 podStartE2EDuration="1.993104922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:32:24.991344928 +0000 UTC m=+903.192387339" watchObservedRunningTime="2026-09-10 16:32:24.993104922 +0000 UTC m=+903.194147371"
Sep 10 16:32:33.854224 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:33.854191    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:33.854621 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:33.854252    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:33.866434 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:33.866410    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:32:34.027548 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:32:34.027512    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:33:03.509003 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:03.508966    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"]
Sep 10 16:33:03.509504 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:03.509236    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9" podUID="4a93ec43-e931-4299-92ab-b092539de479" containerName="payload-processing" containerID="cri-o://6bee92f3221d153b864dcc4b6cde3e9370732f0d99bc55cfe6e7cb3930334361" gracePeriod=30
Sep 10 16:33:03.754745 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:03.754718    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"
Sep 10 16:33:03.831478 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:03.831415    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x9k5\" (UniqueName: \"kubernetes.io/projected/4a93ec43-e931-4299-92ab-b092539de479-kube-api-access-2x9k5\") pod \"4a93ec43-e931-4299-92ab-b092539de479\" (UID: \"4a93ec43-e931-4299-92ab-b092539de479\") "
Sep 10 16:33:03.831478 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:03.831469    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a93ec43-e931-4299-92ab-b092539de479-plugins-config-volume\") pod \"4a93ec43-e931-4299-92ab-b092539de479\" (UID: \"4a93ec43-e931-4299-92ab-b092539de479\") "
Sep 10 16:33:03.831834 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:03.831812    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a93ec43-e931-4299-92ab-b092539de479-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4a93ec43-e931-4299-92ab-b092539de479" (UID: "4a93ec43-e931-4299-92ab-b092539de479"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:33:03.833417 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:03.833391    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a93ec43-e931-4299-92ab-b092539de479-kube-api-access-2x9k5" (OuterVolumeSpecName: "kube-api-access-2x9k5") pod "4a93ec43-e931-4299-92ab-b092539de479" (UID: "4a93ec43-e931-4299-92ab-b092539de479"). InnerVolumeSpecName "kube-api-access-2x9k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:33:03.932067 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:03.932042    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a93ec43-e931-4299-92ab-b092539de479-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:33:03.932067 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:03.932063    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x9k5\" (UniqueName: \"kubernetes.io/projected/4a93ec43-e931-4299-92ab-b092539de479-kube-api-access-2x9k5\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:33:04.151315 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:04.151240    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a93ec43-e931-4299-92ab-b092539de479" containerID="6bee92f3221d153b864dcc4b6cde3e9370732f0d99bc55cfe6e7cb3930334361" exitCode=0
Sep 10 16:33:04.151315 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:04.151283    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"
Sep 10 16:33:04.151462 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:04.151283    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9" event={"ID":"4a93ec43-e931-4299-92ab-b092539de479","Type":"ContainerDied","Data":"6bee92f3221d153b864dcc4b6cde3e9370732f0d99bc55cfe6e7cb3930334361"}
Sep 10 16:33:04.151462 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:04.151413    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9" event={"ID":"4a93ec43-e931-4299-92ab-b092539de479","Type":"ContainerDied","Data":"ac057bfcda2be0312bf600970e6f2a0747f132b958788141e0010c11cc58c182"}
Sep 10 16:33:04.151462 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:04.151438    2580 scope.go:117] "RemoveContainer" containerID="6bee92f3221d153b864dcc4b6cde3e9370732f0d99bc55cfe6e7cb3930334361"
Sep 10 16:33:04.160260 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:04.160243    2580 scope.go:117] "RemoveContainer" containerID="6bee92f3221d153b864dcc4b6cde3e9370732f0d99bc55cfe6e7cb3930334361"
Sep 10 16:33:04.160573 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:33:04.160555    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bee92f3221d153b864dcc4b6cde3e9370732f0d99bc55cfe6e7cb3930334361\": container with ID starting with 6bee92f3221d153b864dcc4b6cde3e9370732f0d99bc55cfe6e7cb3930334361 not found: ID does not exist" containerID="6bee92f3221d153b864dcc4b6cde3e9370732f0d99bc55cfe6e7cb3930334361"
Sep 10 16:33:04.160661 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:04.160585    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bee92f3221d153b864dcc4b6cde3e9370732f0d99bc55cfe6e7cb3930334361"} err="failed to get container status \"6bee92f3221d153b864dcc4b6cde3e9370732f0d99bc55cfe6e7cb3930334361\": rpc error: code = NotFound desc = could not find container \"6bee92f3221d153b864dcc4b6cde3e9370732f0d99bc55cfe6e7cb3930334361\": container with ID starting with 6bee92f3221d153b864dcc4b6cde3e9370732f0d99bc55cfe6e7cb3930334361 not found: ID does not exist"
Sep 10 16:33:04.172978 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:04.172954    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"]
Sep 10 16:33:04.174474 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:04.174456    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-09da25-6847b57d97-tkbf9"]
Sep 10 16:33:04.254796 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:04.254771    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a93ec43-e931-4299-92ab-b092539de479" path="/var/lib/kubelet/pods/4a93ec43-e931-4299-92ab-b092539de479/volumes"
Sep 10 16:33:47.631859 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:47.631785    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"]
Sep 10 16:33:48.747200 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:48.747171    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"]
Sep 10 16:33:59.601337 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:59.601306    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1abdaf-29817630-8jgcc"]
Sep 10 16:33:59.602935 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:59.602911    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1abdaf-29817630-8jgcc"]
Sep 10 16:33:59.625379 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:59.625341    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"]
Sep 10 16:33:59.625680 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:59.625636    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb" podUID="ad4c5804-56fa-43ad-a804-82a72acb32b3" containerName="payload-processing" containerID="cri-o://2035c64db9fa08979e1feae139fce96aeef06591f4a911283f552344d607c3d7" gracePeriod=30
Sep 10 16:33:59.634312 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:59.634288    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"]
Sep 10 16:33:59.634591 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:59.634498    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk" podUID="322d822c-ffba-4a59-a920-64079c784880" containerName="payload-pre-processing" containerID="cri-o://ed74d70cfdcb245ba30954c99ea2b3afa0e7b83090b6f7cea731ecdeb559a329" gracePeriod=30
Sep 10 16:33:59.917107 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:59.917084    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"
Sep 10 16:33:59.920256 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:59.920235    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"
Sep 10 16:33:59.973647 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:59.973620    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9sw9\" (UniqueName: \"kubernetes.io/projected/ad4c5804-56fa-43ad-a804-82a72acb32b3-kube-api-access-f9sw9\") pod \"ad4c5804-56fa-43ad-a804-82a72acb32b3\" (UID: \"ad4c5804-56fa-43ad-a804-82a72acb32b3\") "
Sep 10 16:33:59.973788 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:59.973735    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4c5804-56fa-43ad-a804-82a72acb32b3-plugins-config-volume\") pod \"ad4c5804-56fa-43ad-a804-82a72acb32b3\" (UID: \"ad4c5804-56fa-43ad-a804-82a72acb32b3\") "
Sep 10 16:33:59.974077 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:59.974054    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4c5804-56fa-43ad-a804-82a72acb32b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad4c5804-56fa-43ad-a804-82a72acb32b3" (UID: "ad4c5804-56fa-43ad-a804-82a72acb32b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:33:59.975771 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:33:59.975750    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4c5804-56fa-43ad-a804-82a72acb32b3-kube-api-access-f9sw9" (OuterVolumeSpecName: "kube-api-access-f9sw9") pod "ad4c5804-56fa-43ad-a804-82a72acb32b3" (UID: "ad4c5804-56fa-43ad-a804-82a72acb32b3"). InnerVolumeSpecName "kube-api-access-f9sw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:34:00.074291 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.074271    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322d822c-ffba-4a59-a920-64079c784880-plugins-config-volume\") pod \"322d822c-ffba-4a59-a920-64079c784880\" (UID: \"322d822c-ffba-4a59-a920-64079c784880\") "
Sep 10 16:34:00.074418 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.074305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25nhf\" (UniqueName: \"kubernetes.io/projected/322d822c-ffba-4a59-a920-64079c784880-kube-api-access-25nhf\") pod \"322d822c-ffba-4a59-a920-64079c784880\" (UID: \"322d822c-ffba-4a59-a920-64079c784880\") "
Sep 10 16:34:00.074510 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.074494    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4c5804-56fa-43ad-a804-82a72acb32b3-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:00.074573 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.074517    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9sw9\" (UniqueName: \"kubernetes.io/projected/ad4c5804-56fa-43ad-a804-82a72acb32b3-kube-api-access-f9sw9\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:00.074636 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.074615    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322d822c-ffba-4a59-a920-64079c784880-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "322d822c-ffba-4a59-a920-64079c784880" (UID: "322d822c-ffba-4a59-a920-64079c784880"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:34:00.076335 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.076309    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322d822c-ffba-4a59-a920-64079c784880-kube-api-access-25nhf" (OuterVolumeSpecName: "kube-api-access-25nhf") pod "322d822c-ffba-4a59-a920-64079c784880" (UID: "322d822c-ffba-4a59-a920-64079c784880"). InnerVolumeSpecName "kube-api-access-25nhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:34:00.175034 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.174969    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322d822c-ffba-4a59-a920-64079c784880-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:00.175034 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.174997    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25nhf\" (UniqueName: \"kubernetes.io/projected/322d822c-ffba-4a59-a920-64079c784880-kube-api-access-25nhf\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:00.254474 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.254441    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be3a5461-0fcb-4eb1-a98a-bf5d3e1a84ab" path="/var/lib/kubelet/pods/be3a5461-0fcb-4eb1-a98a-bf5d3e1a84ab/volumes"
Sep 10 16:34:00.400687 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.400656    2580 generic.go:356] "Generic (PLEG): container finished" podID="322d822c-ffba-4a59-a920-64079c784880" containerID="ed74d70cfdcb245ba30954c99ea2b3afa0e7b83090b6f7cea731ecdeb559a329" exitCode=0
Sep 10 16:34:00.400821 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.400699    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"
Sep 10 16:34:00.400821 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.400733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk" event={"ID":"322d822c-ffba-4a59-a920-64079c784880","Type":"ContainerDied","Data":"ed74d70cfdcb245ba30954c99ea2b3afa0e7b83090b6f7cea731ecdeb559a329"}
Sep 10 16:34:00.400821 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.400781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk" event={"ID":"322d822c-ffba-4a59-a920-64079c784880","Type":"ContainerDied","Data":"3ca6f323be0ec61b9058082eae46dd41280f04918f700bbef0ede9932ad5ab30"}
Sep 10 16:34:00.400821 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.400808    2580 scope.go:117] "RemoveContainer" containerID="ed74d70cfdcb245ba30954c99ea2b3afa0e7b83090b6f7cea731ecdeb559a329"
Sep 10 16:34:00.402140 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.402115    2580 generic.go:356] "Generic (PLEG): container finished" podID="ad4c5804-56fa-43ad-a804-82a72acb32b3" containerID="2035c64db9fa08979e1feae139fce96aeef06591f4a911283f552344d607c3d7" exitCode=0
Sep 10 16:34:00.402266 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.402161    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"
Sep 10 16:34:00.402266 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.402180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb" event={"ID":"ad4c5804-56fa-43ad-a804-82a72acb32b3","Type":"ContainerDied","Data":"2035c64db9fa08979e1feae139fce96aeef06591f4a911283f552344d607c3d7"}
Sep 10 16:34:00.402266 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.402231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb" event={"ID":"ad4c5804-56fa-43ad-a804-82a72acb32b3","Type":"ContainerDied","Data":"13cd1487bac57cd50deeef5d2e603df1cdf16325b472e28a59e746fe3e057b36"}
Sep 10 16:34:00.409430 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.409409    2580 scope.go:117] "RemoveContainer" containerID="ed74d70cfdcb245ba30954c99ea2b3afa0e7b83090b6f7cea731ecdeb559a329"
Sep 10 16:34:00.409756 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:34:00.409731    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed74d70cfdcb245ba30954c99ea2b3afa0e7b83090b6f7cea731ecdeb559a329\": container with ID starting with ed74d70cfdcb245ba30954c99ea2b3afa0e7b83090b6f7cea731ecdeb559a329 not found: ID does not exist" containerID="ed74d70cfdcb245ba30954c99ea2b3afa0e7b83090b6f7cea731ecdeb559a329"
Sep 10 16:34:00.409835 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.409763    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed74d70cfdcb245ba30954c99ea2b3afa0e7b83090b6f7cea731ecdeb559a329"} err="failed to get container status \"ed74d70cfdcb245ba30954c99ea2b3afa0e7b83090b6f7cea731ecdeb559a329\": rpc error: code = NotFound desc = could not find container \"ed74d70cfdcb245ba30954c99ea2b3afa0e7b83090b6f7cea731ecdeb559a329\": container with ID starting with ed74d70cfdcb245ba30954c99ea2b3afa0e7b83090b6f7cea731ecdeb559a329 not found: ID does not exist"
Sep 10 16:34:00.409835 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.409786    2580 scope.go:117] "RemoveContainer" containerID="2035c64db9fa08979e1feae139fce96aeef06591f4a911283f552344d607c3d7"
Sep 10 16:34:00.417518 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.417502    2580 scope.go:117] "RemoveContainer" containerID="2035c64db9fa08979e1feae139fce96aeef06591f4a911283f552344d607c3d7"
Sep 10 16:34:00.417810 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:34:00.417794    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2035c64db9fa08979e1feae139fce96aeef06591f4a911283f552344d607c3d7\": container with ID starting with 2035c64db9fa08979e1feae139fce96aeef06591f4a911283f552344d607c3d7 not found: ID does not exist" containerID="2035c64db9fa08979e1feae139fce96aeef06591f4a911283f552344d607c3d7"
Sep 10 16:34:00.417858 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.417820    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2035c64db9fa08979e1feae139fce96aeef06591f4a911283f552344d607c3d7"} err="failed to get container status \"2035c64db9fa08979e1feae139fce96aeef06591f4a911283f552344d607c3d7\": rpc error: code = NotFound desc = could not find container \"2035c64db9fa08979e1feae139fce96aeef06591f4a911283f552344d607c3d7\": container with ID starting with 2035c64db9fa08979e1feae139fce96aeef06591f4a911283f552344d607c3d7 not found: ID does not exist"
Sep 10 16:34:00.425493 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.425447    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"]
Sep 10 16:34:00.430863 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.430845    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1abdaf-d7d4666c5-gwdjb"]
Sep 10 16:34:00.440343 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.440325    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"]
Sep 10 16:34:00.445812 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:00.445795    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1abdaf-5bf98b995d-4dqbk"]
Sep 10 16:34:02.253885 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.253850    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="322d822c-ffba-4a59-a920-64079c784880" path="/var/lib/kubelet/pods/322d822c-ffba-4a59-a920-64079c784880/volumes"
Sep 10 16:34:02.254310 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.254235    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad4c5804-56fa-43ad-a804-82a72acb32b3" path="/var/lib/kubelet/pods/ad4c5804-56fa-43ad-a804-82a72acb32b3/volumes"
Sep 10 16:34:02.651140 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.651107    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s" podUID="3125dadf-c703-4ab0-aa03-322c6de2b827" containerName="main" containerID="cri-o://7b5aeb825a43c3c973d81c3b8ff48babd5c496bbb09060eb55efdc5b8bb45d74" gracePeriod=45
Sep 10 16:34:02.901214 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.901161    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:34:02.995719 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.995689    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-dshm\") pod \"3125dadf-c703-4ab0-aa03-322c6de2b827\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") "
Sep 10 16:34:02.995833 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.995727    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk5t7\" (UniqueName: \"kubernetes.io/projected/3125dadf-c703-4ab0-aa03-322c6de2b827-kube-api-access-wk5t7\") pod \"3125dadf-c703-4ab0-aa03-322c6de2b827\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") "
Sep 10 16:34:02.995833 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.995791    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3125dadf-c703-4ab0-aa03-322c6de2b827-tls-certs\") pod \"3125dadf-c703-4ab0-aa03-322c6de2b827\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") "
Sep 10 16:34:02.995903 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.995842    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-home\") pod \"3125dadf-c703-4ab0-aa03-322c6de2b827\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") "
Sep 10 16:34:02.995903 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.995887    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-tmp-dir\") pod \"3125dadf-c703-4ab0-aa03-322c6de2b827\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") "
Sep 10 16:34:02.995987 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.995915    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-model-cache\") pod \"3125dadf-c703-4ab0-aa03-322c6de2b827\" (UID: \"3125dadf-c703-4ab0-aa03-322c6de2b827\") "
Sep 10 16:34:02.996192 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.996150    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-home" (OuterVolumeSpecName: "home") pod "3125dadf-c703-4ab0-aa03-322c6de2b827" (UID: "3125dadf-c703-4ab0-aa03-322c6de2b827"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:34:02.996309 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.996184    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3125dadf-c703-4ab0-aa03-322c6de2b827" (UID: "3125dadf-c703-4ab0-aa03-322c6de2b827"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:34:02.996309 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.996261    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-model-cache" (OuterVolumeSpecName: "model-cache") pod "3125dadf-c703-4ab0-aa03-322c6de2b827" (UID: "3125dadf-c703-4ab0-aa03-322c6de2b827"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:34:02.997895 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.997873    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-dshm" (OuterVolumeSpecName: "dshm") pod "3125dadf-c703-4ab0-aa03-322c6de2b827" (UID: "3125dadf-c703-4ab0-aa03-322c6de2b827"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:34:02.997985 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.997914    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3125dadf-c703-4ab0-aa03-322c6de2b827-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3125dadf-c703-4ab0-aa03-322c6de2b827" (UID: "3125dadf-c703-4ab0-aa03-322c6de2b827"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:34:02.998030 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:02.997995    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3125dadf-c703-4ab0-aa03-322c6de2b827-kube-api-access-wk5t7" (OuterVolumeSpecName: "kube-api-access-wk5t7") pod "3125dadf-c703-4ab0-aa03-322c6de2b827" (UID: "3125dadf-c703-4ab0-aa03-322c6de2b827"). InnerVolumeSpecName "kube-api-access-wk5t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:34:03.097474 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.097452    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk5t7\" (UniqueName: \"kubernetes.io/projected/3125dadf-c703-4ab0-aa03-322c6de2b827-kube-api-access-wk5t7\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:03.097474 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.097472    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3125dadf-c703-4ab0-aa03-322c6de2b827-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:03.097623 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.097482    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:03.097623 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.097490    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:03.097623 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.097497    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:03.097623 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.097504    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3125dadf-c703-4ab0-aa03-322c6de2b827-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:03.419887 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.419858    2580 generic.go:356] "Generic (PLEG): container finished" podID="3125dadf-c703-4ab0-aa03-322c6de2b827" containerID="7b5aeb825a43c3c973d81c3b8ff48babd5c496bbb09060eb55efdc5b8bb45d74" exitCode=0
Sep 10 16:34:03.420344 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.419907    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"
Sep 10 16:34:03.420344 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.419938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s" event={"ID":"3125dadf-c703-4ab0-aa03-322c6de2b827","Type":"ContainerDied","Data":"7b5aeb825a43c3c973d81c3b8ff48babd5c496bbb09060eb55efdc5b8bb45d74"}
Sep 10 16:34:03.420344 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.419981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s" event={"ID":"3125dadf-c703-4ab0-aa03-322c6de2b827","Type":"ContainerDied","Data":"43372ce72b808b92d258a989f71a5a165dbafb88e655b4edcb4d1a372a6458f7"}
Sep 10 16:34:03.420344 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.419998    2580 scope.go:117] "RemoveContainer" containerID="7b5aeb825a43c3c973d81c3b8ff48babd5c496bbb09060eb55efdc5b8bb45d74"
Sep 10 16:34:03.428406 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.428388    2580 scope.go:117] "RemoveContainer" containerID="7b5aeb825a43c3c973d81c3b8ff48babd5c496bbb09060eb55efdc5b8bb45d74"
Sep 10 16:34:03.428704 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:34:03.428687    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5aeb825a43c3c973d81c3b8ff48babd5c496bbb09060eb55efdc5b8bb45d74\": container with ID starting with 7b5aeb825a43c3c973d81c3b8ff48babd5c496bbb09060eb55efdc5b8bb45d74 not found: ID does not exist" containerID="7b5aeb825a43c3c973d81c3b8ff48babd5c496bbb09060eb55efdc5b8bb45d74"
Sep 10 16:34:03.428755 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.428716    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5aeb825a43c3c973d81c3b8ff48babd5c496bbb09060eb55efdc5b8bb45d74"} err="failed to get container status \"7b5aeb825a43c3c973d81c3b8ff48babd5c496bbb09060eb55efdc5b8bb45d74\": rpc error: code = NotFound desc = could not find container \"7b5aeb825a43c3c973d81c3b8ff48babd5c496bbb09060eb55efdc5b8bb45d74\": container with ID starting with 7b5aeb825a43c3c973d81c3b8ff48babd5c496bbb09060eb55efdc5b8bb45d74 not found: ID does not exist"
Sep 10 16:34:03.439424 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.439403    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"]
Sep 10 16:34:03.440883 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.440860    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1abdaf/test-model-1abdaf-kserve-f754fb599-hwr9s"]
Sep 10 16:34:03.766576 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:03.766545    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg" podUID="616050b2-189e-4b56-a049-314f73837e68" containerName="main" containerID="cri-o://fb1a24e25083c43ad8e5413b4bc4f59bec595670592f1999dfccbb5f3e90b9d1" gracePeriod=45
Sep 10 16:34:04.022926 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.022874    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:34:04.105831 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.105805    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-home\") pod \"616050b2-189e-4b56-a049-314f73837e68\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") "
Sep 10 16:34:04.105967 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.105867    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616050b2-189e-4b56-a049-314f73837e68-tls-certs\") pod \"616050b2-189e-4b56-a049-314f73837e68\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") "
Sep 10 16:34:04.105967 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.105903    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg746\" (UniqueName: \"kubernetes.io/projected/616050b2-189e-4b56-a049-314f73837e68-kube-api-access-tg746\") pod \"616050b2-189e-4b56-a049-314f73837e68\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") "
Sep 10 16:34:04.105967 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.105937    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-tmp-dir\") pod \"616050b2-189e-4b56-a049-314f73837e68\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") "
Sep 10 16:34:04.106125 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.105969    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-model-cache\") pod \"616050b2-189e-4b56-a049-314f73837e68\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") "
Sep 10 16:34:04.106125 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.106014    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-dshm\") pod \"616050b2-189e-4b56-a049-314f73837e68\" (UID: \"616050b2-189e-4b56-a049-314f73837e68\") "
Sep 10 16:34:04.106125 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.106035    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-home" (OuterVolumeSpecName: "home") pod "616050b2-189e-4b56-a049-314f73837e68" (UID: "616050b2-189e-4b56-a049-314f73837e68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:34:04.106316 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.106223    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-model-cache" (OuterVolumeSpecName: "model-cache") pod "616050b2-189e-4b56-a049-314f73837e68" (UID: "616050b2-189e-4b56-a049-314f73837e68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:34:04.106316 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.106239    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "616050b2-189e-4b56-a049-314f73837e68" (UID: "616050b2-189e-4b56-a049-314f73837e68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:34:04.106316 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.106279    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:04.108090 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.108066    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-dshm" (OuterVolumeSpecName: "dshm") pod "616050b2-189e-4b56-a049-314f73837e68" (UID: "616050b2-189e-4b56-a049-314f73837e68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:34:04.108201 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.108134    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/616050b2-189e-4b56-a049-314f73837e68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "616050b2-189e-4b56-a049-314f73837e68" (UID: "616050b2-189e-4b56-a049-314f73837e68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:34:04.108201 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.108154    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616050b2-189e-4b56-a049-314f73837e68-kube-api-access-tg746" (OuterVolumeSpecName: "kube-api-access-tg746") pod "616050b2-189e-4b56-a049-314f73837e68" (UID: "616050b2-189e-4b56-a049-314f73837e68"). InnerVolumeSpecName "kube-api-access-tg746". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:34:04.207394 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.207362    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616050b2-189e-4b56-a049-314f73837e68-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:04.207394 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.207393    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg746\" (UniqueName: \"kubernetes.io/projected/616050b2-189e-4b56-a049-314f73837e68-kube-api-access-tg746\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:04.207394 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.207401    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:04.207574 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.207412    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:04.207574 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.207420    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616050b2-189e-4b56-a049-314f73837e68-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:04.255593 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.255558    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3125dadf-c703-4ab0-aa03-322c6de2b827" path="/var/lib/kubelet/pods/3125dadf-c703-4ab0-aa03-322c6de2b827/volumes"
Sep 10 16:34:04.426450 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.426392    2580 generic.go:356] "Generic (PLEG): container finished" podID="616050b2-189e-4b56-a049-314f73837e68" containerID="fb1a24e25083c43ad8e5413b4bc4f59bec595670592f1999dfccbb5f3e90b9d1" exitCode=0
Sep 10 16:34:04.426450 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.426434    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"
Sep 10 16:34:04.426865 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.426480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg" event={"ID":"616050b2-189e-4b56-a049-314f73837e68","Type":"ContainerDied","Data":"fb1a24e25083c43ad8e5413b4bc4f59bec595670592f1999dfccbb5f3e90b9d1"}
Sep 10 16:34:04.426865 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.426513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg" event={"ID":"616050b2-189e-4b56-a049-314f73837e68","Type":"ContainerDied","Data":"65bcc42bc067799b53dc9863d010a4e24eb9527c22fc6803b8d04f54b103b0cf"}
Sep 10 16:34:04.426865 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.426555    2580 scope.go:117] "RemoveContainer" containerID="fb1a24e25083c43ad8e5413b4bc4f59bec595670592f1999dfccbb5f3e90b9d1"
Sep 10 16:34:04.434676 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.434651    2580 scope.go:117] "RemoveContainer" containerID="fb1a24e25083c43ad8e5413b4bc4f59bec595670592f1999dfccbb5f3e90b9d1"
Sep 10 16:34:04.434930 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:34:04.434911    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1a24e25083c43ad8e5413b4bc4f59bec595670592f1999dfccbb5f3e90b9d1\": container with ID starting with fb1a24e25083c43ad8e5413b4bc4f59bec595670592f1999dfccbb5f3e90b9d1 not found: ID does not exist" containerID="fb1a24e25083c43ad8e5413b4bc4f59bec595670592f1999dfccbb5f3e90b9d1"
Sep 10 16:34:04.434976 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.434937    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1a24e25083c43ad8e5413b4bc4f59bec595670592f1999dfccbb5f3e90b9d1"} err="failed to get container status \"fb1a24e25083c43ad8e5413b4bc4f59bec595670592f1999dfccbb5f3e90b9d1\": rpc error: code = NotFound desc = could not find container \"fb1a24e25083c43ad8e5413b4bc4f59bec595670592f1999dfccbb5f3e90b9d1\": container with ID starting with fb1a24e25083c43ad8e5413b4bc4f59bec595670592f1999dfccbb5f3e90b9d1 not found: ID does not exist"
Sep 10 16:34:04.442234 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.442213    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"]
Sep 10 16:34:04.447492 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:04.447472    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg"]
Sep 10 16:34:06.254894 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:06.254859    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="616050b2-189e-4b56-a049-314f73837e68" path="/var/lib/kubelet/pods/616050b2-189e-4b56-a049-314f73837e68/volumes"
Sep 10 16:34:07.192442 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192410    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf"]
Sep 10 16:34:07.192730 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192717    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3125dadf-c703-4ab0-aa03-322c6de2b827" containerName="main"
Sep 10 16:34:07.192779 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192731    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3125dadf-c703-4ab0-aa03-322c6de2b827" containerName="main"
Sep 10 16:34:07.192779 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192740    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad4c5804-56fa-43ad-a804-82a72acb32b3" containerName="payload-processing"
Sep 10 16:34:07.192779 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192746    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad4c5804-56fa-43ad-a804-82a72acb32b3" containerName="payload-processing"
Sep 10 16:34:07.192779 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192753    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a93ec43-e931-4299-92ab-b092539de479" containerName="payload-processing"
Sep 10 16:34:07.192779 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192759    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a93ec43-e931-4299-92ab-b092539de479" containerName="payload-processing"
Sep 10 16:34:07.192779 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192766    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="616050b2-189e-4b56-a049-314f73837e68" containerName="main"
Sep 10 16:34:07.192779 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192771    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="616050b2-189e-4b56-a049-314f73837e68" containerName="main"
Sep 10 16:34:07.192779 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192782    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322d822c-ffba-4a59-a920-64079c784880" containerName="payload-pre-processing"
Sep 10 16:34:07.193005 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192787    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="322d822c-ffba-4a59-a920-64079c784880" containerName="payload-pre-processing"
Sep 10 16:34:07.193005 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192836    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="322d822c-ffba-4a59-a920-64079c784880" containerName="payload-pre-processing"
Sep 10 16:34:07.193005 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192843    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad4c5804-56fa-43ad-a804-82a72acb32b3" containerName="payload-processing"
Sep 10 16:34:07.193005 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192849    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="616050b2-189e-4b56-a049-314f73837e68" containerName="main"
Sep 10 16:34:07.193005 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192855    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3125dadf-c703-4ab0-aa03-322c6de2b827" containerName="main"
Sep 10 16:34:07.193005 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.192860    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a93ec43-e931-4299-92ab-b092539de479" containerName="payload-processing"
Sep 10 16:34:07.197263 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.197247    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf"
Sep 10 16:34:07.199430 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.199408    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\""
Sep 10 16:34:07.199564 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.199436    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:34:07.202180 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.202162    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf"]
Sep 10 16:34:07.333968 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.333938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c203a8c9-b7ac-479a-9621-8d8506872338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf\" (UID: \"c203a8c9-b7ac-479a-9621-8d8506872338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf"
Sep 10 16:34:07.434967 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.434940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c203a8c9-b7ac-479a-9621-8d8506872338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf\" (UID: \"c203a8c9-b7ac-479a-9621-8d8506872338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf"
Sep 10 16:34:07.435511 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.435492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c203a8c9-b7ac-479a-9621-8d8506872338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf\" (UID: \"c203a8c9-b7ac-479a-9621-8d8506872338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf"
Sep 10 16:34:07.509155 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.509135    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf"
Sep 10 16:34:07.775107 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:07.775082    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf"]
Sep 10 16:34:07.777106 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:34:07.777072    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc203a8c9_b7ac_479a_9621_8d8506872338.slice/crio-1915d0ce68c57470abd1f0193a1017a1d474c9110d85bd0548a17c61f434257a WatchSource:0}: Error finding container 1915d0ce68c57470abd1f0193a1017a1d474c9110d85bd0548a17c61f434257a: Status 404 returned error can't find the container with id 1915d0ce68c57470abd1f0193a1017a1d474c9110d85bd0548a17c61f434257a
Sep 10 16:34:08.447278 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:08.447254    2580 generic.go:356] "Generic (PLEG): container finished" podID="c203a8c9-b7ac-479a-9621-8d8506872338" containerID="cc137dd849f8a94f2c40ec44e1919d11b4594758bd448ab13af39428553d4c4e" exitCode=0
Sep 10 16:34:08.447589 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:08.447340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf" event={"ID":"c203a8c9-b7ac-479a-9621-8d8506872338","Type":"ContainerDied","Data":"cc137dd849f8a94f2c40ec44e1919d11b4594758bd448ab13af39428553d4c4e"}
Sep 10 16:34:08.447589 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:08.447390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf" event={"ID":"c203a8c9-b7ac-479a-9621-8d8506872338","Type":"ContainerStarted","Data":"1915d0ce68c57470abd1f0193a1017a1d474c9110d85bd0548a17c61f434257a"}
Sep 10 16:34:09.017360 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:09.017321    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9f6082/test-model-9f6082-kserve-79c74867c9-pr7vg" podUID="616050b2-189e-4b56-a049-314f73837e68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 10 16:34:09.572201 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:09.572180    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf"
Sep 10 16:34:09.651505 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:09.651481    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c203a8c9-b7ac-479a-9621-8d8506872338-maas-api-service-ca\") pod \"c203a8c9-b7ac-479a-9621-8d8506872338\" (UID: \"c203a8c9-b7ac-479a-9621-8d8506872338\") "
Sep 10 16:34:09.651829 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:09.651805    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c203a8c9-b7ac-479a-9621-8d8506872338-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c203a8c9-b7ac-479a-9621-8d8506872338" (UID: "c203a8c9-b7ac-479a-9621-8d8506872338"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:34:09.752879 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:09.752824    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c203a8c9-b7ac-479a-9621-8d8506872338-maas-api-service-ca\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:10.457398 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:10.457370    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf"
Sep 10 16:34:10.457563 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:10.457395    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf" event={"ID":"c203a8c9-b7ac-479a-9621-8d8506872338","Type":"ContainerDied","Data":"1915d0ce68c57470abd1f0193a1017a1d474c9110d85bd0548a17c61f434257a"}
Sep 10 16:34:10.457563 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:10.457438    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1915d0ce68c57470abd1f0193a1017a1d474c9110d85bd0548a17c61f434257a"
Sep 10 16:34:17.284446 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.284409    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2"]
Sep 10 16:34:17.284810 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.284766    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c203a8c9-b7ac-479a-9621-8d8506872338" containerName="revoke-keys"
Sep 10 16:34:17.284810 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.284779    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c203a8c9-b7ac-479a-9621-8d8506872338" containerName="revoke-keys"
Sep 10 16:34:17.284888 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.284829    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c203a8c9-b7ac-479a-9621-8d8506872338" containerName="revoke-keys"
Sep 10 16:34:17.287707 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.287683    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2"
Sep 10 16:34:17.290016 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.289994    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:34:17.290151 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.289996    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\""
Sep 10 16:34:17.296237 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.296216    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2"]
Sep 10 16:34:17.409518 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.409478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4b62db-73b8-4b94-a5b3-9b405460404d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2\" (UID: \"fa4b62db-73b8-4b94-a5b3-9b405460404d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2"
Sep 10 16:34:17.510386 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.510350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4b62db-73b8-4b94-a5b3-9b405460404d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2\" (UID: \"fa4b62db-73b8-4b94-a5b3-9b405460404d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2"
Sep 10 16:34:17.510974 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.510955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4b62db-73b8-4b94-a5b3-9b405460404d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2\" (UID: \"fa4b62db-73b8-4b94-a5b3-9b405460404d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2"
Sep 10 16:34:17.598873 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.598793    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2"
Sep 10 16:34:17.866157 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.866133    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2"]
Sep 10 16:34:17.868463 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:34:17.868437    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4b62db_73b8_4b94_a5b3_9b405460404d.slice/crio-f7ae658386e0418b1a59b63220885182fe7053178e5732aa27485835a42cc695 WatchSource:0}: Error finding container f7ae658386e0418b1a59b63220885182fe7053178e5732aa27485835a42cc695: Status 404 returned error can't find the container with id f7ae658386e0418b1a59b63220885182fe7053178e5732aa27485835a42cc695
Sep 10 16:34:17.991272 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.991240    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"]
Sep 10 16:34:17.991501 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:17.991474    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9" podUID="e89656b6-d9d2-4cd1-b742-1ed8e2468f94" containerName="payload-processing" containerID="cri-o://9caa9a07e59d05aa0c5a269566b9a46350f4f6827d1171b6a7184c7202dda8c4" gracePeriod=30
Sep 10 16:34:18.001868 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.001844    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"]
Sep 10 16:34:18.002048 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.002029    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6" podUID="4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3" containerName="payload-pre-processing" containerID="cri-o://1febb2a2974e4bae1c1523c3b875d67408771f8fe47933718aba8935ac979332" gracePeriod=30
Sep 10 16:34:18.144245 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.144215    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf"]
Sep 10 16:34:18.148922 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.148814    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6082-k9tdf"]
Sep 10 16:34:18.233397 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.233371    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"
Sep 10 16:34:18.263822 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.263776    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c203a8c9-b7ac-479a-9621-8d8506872338" path="/var/lib/kubelet/pods/c203a8c9-b7ac-479a-9621-8d8506872338/volumes"
Sep 10 16:34:18.272348 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.272323    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"
Sep 10 16:34:18.315431 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.315400    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkx6l\" (UniqueName: \"kubernetes.io/projected/e89656b6-d9d2-4cd1-b742-1ed8e2468f94-kube-api-access-rkx6l\") pod \"e89656b6-d9d2-4cd1-b742-1ed8e2468f94\" (UID: \"e89656b6-d9d2-4cd1-b742-1ed8e2468f94\") "
Sep 10 16:34:18.315782 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.315509    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e89656b6-d9d2-4cd1-b742-1ed8e2468f94-plugins-config-volume\") pod \"e89656b6-d9d2-4cd1-b742-1ed8e2468f94\" (UID: \"e89656b6-d9d2-4cd1-b742-1ed8e2468f94\") "
Sep 10 16:34:18.315851 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.315824    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89656b6-d9d2-4cd1-b742-1ed8e2468f94-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e89656b6-d9d2-4cd1-b742-1ed8e2468f94" (UID: "e89656b6-d9d2-4cd1-b742-1ed8e2468f94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:34:18.317461 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.317439    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89656b6-d9d2-4cd1-b742-1ed8e2468f94-kube-api-access-rkx6l" (OuterVolumeSpecName: "kube-api-access-rkx6l") pod "e89656b6-d9d2-4cd1-b742-1ed8e2468f94" (UID: "e89656b6-d9d2-4cd1-b742-1ed8e2468f94"). InnerVolumeSpecName "kube-api-access-rkx6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:34:18.416602 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.416496    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbcm2\" (UniqueName: \"kubernetes.io/projected/4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3-kube-api-access-qbcm2\") pod \"4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3\" (UID: \"4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3\") "
Sep 10 16:34:18.416755 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.416629    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3-plugins-config-volume\") pod \"4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3\" (UID: \"4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3\") "
Sep 10 16:34:18.416799 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.416766    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e89656b6-d9d2-4cd1-b742-1ed8e2468f94-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:18.416799 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.416779    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkx6l\" (UniqueName: \"kubernetes.io/projected/e89656b6-d9d2-4cd1-b742-1ed8e2468f94-kube-api-access-rkx6l\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:18.416976 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.416948    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3" (UID: "4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:34:18.418517 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.418499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3-kube-api-access-qbcm2" (OuterVolumeSpecName: "kube-api-access-qbcm2") pod "4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3" (UID: "4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3"). InnerVolumeSpecName "kube-api-access-qbcm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:34:18.493881 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.493846    2580 generic.go:356] "Generic (PLEG): container finished" podID="e89656b6-d9d2-4cd1-b742-1ed8e2468f94" containerID="9caa9a07e59d05aa0c5a269566b9a46350f4f6827d1171b6a7184c7202dda8c4" exitCode=0
Sep 10 16:34:18.494045 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.493928    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"
Sep 10 16:34:18.494045 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.493928    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9" event={"ID":"e89656b6-d9d2-4cd1-b742-1ed8e2468f94","Type":"ContainerDied","Data":"9caa9a07e59d05aa0c5a269566b9a46350f4f6827d1171b6a7184c7202dda8c4"}
Sep 10 16:34:18.494045 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.493998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9" event={"ID":"e89656b6-d9d2-4cd1-b742-1ed8e2468f94","Type":"ContainerDied","Data":"68e3536a5aec4be76a3f87cec5df227b806541a72d8c4fa648f2ed16bf1908b8"}
Sep 10 16:34:18.494045 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.494031    2580 scope.go:117] "RemoveContainer" containerID="9caa9a07e59d05aa0c5a269566b9a46350f4f6827d1171b6a7184c7202dda8c4"
Sep 10 16:34:18.495290 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.495269    2580 generic.go:356] "Generic (PLEG): container finished" podID="4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3" containerID="1febb2a2974e4bae1c1523c3b875d67408771f8fe47933718aba8935ac979332" exitCode=0
Sep 10 16:34:18.495393 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.495297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6" event={"ID":"4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3","Type":"ContainerDied","Data":"1febb2a2974e4bae1c1523c3b875d67408771f8fe47933718aba8935ac979332"}
Sep 10 16:34:18.495393 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.495352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6" event={"ID":"4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3","Type":"ContainerDied","Data":"d473335a57c4e85f9390ae9b84966395a342b33774ab2d7d27d94fcb86efb3bd"}
Sep 10 16:34:18.495393 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.495308    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"
Sep 10 16:34:18.497351 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.497320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2" event={"ID":"fa4b62db-73b8-4b94-a5b3-9b405460404d","Type":"ContainerStarted","Data":"cff42a9f1a787c58c2551f8cf2271e68a00328c763545dace6087beead0178ad"}
Sep 10 16:34:18.497463 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.497371    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2" event={"ID":"fa4b62db-73b8-4b94-a5b3-9b405460404d","Type":"ContainerStarted","Data":"f7ae658386e0418b1a59b63220885182fe7053178e5732aa27485835a42cc695"}
Sep 10 16:34:18.504336 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.504321    2580 scope.go:117] "RemoveContainer" containerID="9caa9a07e59d05aa0c5a269566b9a46350f4f6827d1171b6a7184c7202dda8c4"
Sep 10 16:34:18.504866 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:34:18.504711    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9caa9a07e59d05aa0c5a269566b9a46350f4f6827d1171b6a7184c7202dda8c4\": container with ID starting with 9caa9a07e59d05aa0c5a269566b9a46350f4f6827d1171b6a7184c7202dda8c4 not found: ID does not exist" containerID="9caa9a07e59d05aa0c5a269566b9a46350f4f6827d1171b6a7184c7202dda8c4"
Sep 10 16:34:18.504866 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.504767    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9caa9a07e59d05aa0c5a269566b9a46350f4f6827d1171b6a7184c7202dda8c4"} err="failed to get container status \"9caa9a07e59d05aa0c5a269566b9a46350f4f6827d1171b6a7184c7202dda8c4\": rpc error: code = NotFound desc = could not find container \"9caa9a07e59d05aa0c5a269566b9a46350f4f6827d1171b6a7184c7202dda8c4\": container with ID starting with 9caa9a07e59d05aa0c5a269566b9a46350f4f6827d1171b6a7184c7202dda8c4 not found: ID does not exist"
Sep 10 16:34:18.504866 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.504793    2580 scope.go:117] "RemoveContainer" containerID="1febb2a2974e4bae1c1523c3b875d67408771f8fe47933718aba8935ac979332"
Sep 10 16:34:18.512748 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.512728    2580 scope.go:117] "RemoveContainer" containerID="1febb2a2974e4bae1c1523c3b875d67408771f8fe47933718aba8935ac979332"
Sep 10 16:34:18.513004 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:34:18.512985    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1febb2a2974e4bae1c1523c3b875d67408771f8fe47933718aba8935ac979332\": container with ID starting with 1febb2a2974e4bae1c1523c3b875d67408771f8fe47933718aba8935ac979332 not found: ID does not exist" containerID="1febb2a2974e4bae1c1523c3b875d67408771f8fe47933718aba8935ac979332"
Sep 10 16:34:18.513073 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.513014    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1febb2a2974e4bae1c1523c3b875d67408771f8fe47933718aba8935ac979332"} err="failed to get container status \"1febb2a2974e4bae1c1523c3b875d67408771f8fe47933718aba8935ac979332\": rpc error: code = NotFound desc = could not find container \"1febb2a2974e4bae1c1523c3b875d67408771f8fe47933718aba8935ac979332\": container with ID starting with 1febb2a2974e4bae1c1523c3b875d67408771f8fe47933718aba8935ac979332 not found: ID does not exist"
Sep 10 16:34:18.515000 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.514965    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2" podStartSLOduration=1.5149534500000001 podStartE2EDuration="1.51495345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:34:18.513998957 +0000 UTC m=+1016.715041378" watchObservedRunningTime="2026-09-10 16:34:18.51495345 +0000 UTC m=+1016.715995860"
Sep 10 16:34:18.517120 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.517099    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:18.517198 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.517124    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbcm2\" (UniqueName: \"kubernetes.io/projected/4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3-kube-api-access-qbcm2\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:18.525390 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.525369    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"]
Sep 10 16:34:18.531018 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.531000    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f6082-fd4dbcddb-f8fg6"]
Sep 10 16:34:18.543428 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.543408    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"]
Sep 10 16:34:18.547316 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:18.547297    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f6082-648fd668d5-jqfd9"]
Sep 10 16:34:19.505081 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:19.505036    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa4b62db-73b8-4b94-a5b3-9b405460404d" containerID="cff42a9f1a787c58c2551f8cf2271e68a00328c763545dace6087beead0178ad" exitCode=0
Sep 10 16:34:19.505426 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:19.505103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2" event={"ID":"fa4b62db-73b8-4b94-a5b3-9b405460404d","Type":"ContainerDied","Data":"cff42a9f1a787c58c2551f8cf2271e68a00328c763545dace6087beead0178ad"}
Sep 10 16:34:20.253914 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:20.253884    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3" path="/var/lib/kubelet/pods/4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3/volumes"
Sep 10 16:34:20.254226 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:20.254212    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e89656b6-d9d2-4cd1-b742-1ed8e2468f94" path="/var/lib/kubelet/pods/e89656b6-d9d2-4cd1-b742-1ed8e2468f94/volumes"
Sep 10 16:34:20.631777 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:20.631754    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2"
Sep 10 16:34:20.733760 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:20.733733    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4b62db-73b8-4b94-a5b3-9b405460404d-maas-api-service-ca\") pod \"fa4b62db-73b8-4b94-a5b3-9b405460404d\" (UID: \"fa4b62db-73b8-4b94-a5b3-9b405460404d\") "
Sep 10 16:34:20.734080 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:20.734057    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4b62db-73b8-4b94-a5b3-9b405460404d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa4b62db-73b8-4b94-a5b3-9b405460404d" (UID: "fa4b62db-73b8-4b94-a5b3-9b405460404d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:34:20.834817 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:20.834757    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4b62db-73b8-4b94-a5b3-9b405460404d-maas-api-service-ca\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:34:21.517957 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:21.517920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2" event={"ID":"fa4b62db-73b8-4b94-a5b3-9b405460404d","Type":"ContainerDied","Data":"f7ae658386e0418b1a59b63220885182fe7053178e5732aa27485835a42cc695"}
Sep 10 16:34:21.518123 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:21.517985    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7ae658386e0418b1a59b63220885182fe7053178e5732aa27485835a42cc695"
Sep 10 16:34:21.518123 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:21.517931    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2"
Sep 10 16:34:27.223491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:27.223455    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2"]
Sep 10 16:34:27.225975 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:27.225944    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e77069-wn5k2"]
Sep 10 16:34:28.255549 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:28.255491    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa4b62db-73b8-4b94-a5b3-9b405460404d" path="/var/lib/kubelet/pods/fa4b62db-73b8-4b94-a5b3-9b405460404d/volumes"
Sep 10 16:34:30.879081 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:30.879046    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"]
Sep 10 16:34:30.879497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:30.879354    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89656b6-d9d2-4cd1-b742-1ed8e2468f94" containerName="payload-processing"
Sep 10 16:34:30.879497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:30.879365    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89656b6-d9d2-4cd1-b742-1ed8e2468f94" containerName="payload-processing"
Sep 10 16:34:30.879497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:30.879386    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3" containerName="payload-pre-processing"
Sep 10 16:34:30.879497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:30.879394    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3" containerName="payload-pre-processing"
Sep 10 16:34:30.879497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:30.879408    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4b62db-73b8-4b94-a5b3-9b405460404d" containerName="revoke-keys"
Sep 10 16:34:30.879497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:30.879414    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4b62db-73b8-4b94-a5b3-9b405460404d" containerName="revoke-keys"
Sep 10 16:34:30.879497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:30.879460    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4afdc4f6-a5b8-4ad8-9d18-e4724edc61c3" containerName="payload-pre-processing"
Sep 10 16:34:30.879497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:30.879471    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89656b6-d9d2-4cd1-b742-1ed8e2468f94" containerName="payload-processing"
Sep 10 16:34:30.879497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:30.879479    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4b62db-73b8-4b94-a5b3-9b405460404d" containerName="revoke-keys"
Sep 10 16:34:30.883886 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:30.883869    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"
Sep 10 16:34:30.886162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:30.886139    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1925d8-dockercfg-9n7gq\""
Sep 10 16:34:30.886294 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:30.886235    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1925d8\""
Sep 10 16:34:30.892242 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:30.892223    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"]
Sep 10 16:34:31.014906 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:31.014870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c4wf\" (UniqueName: \"kubernetes.io/projected/739e13de-b90f-434d-805d-e75b1e6c9b2e-kube-api-access-2c4wf\") pod \"payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks\" (UID: \"739e13de-b90f-434d-805d-e75b1e6c9b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"
Sep 10 16:34:31.015063 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:31.014981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739e13de-b90f-434d-805d-e75b1e6c9b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks\" (UID: \"739e13de-b90f-434d-805d-e75b1e6c9b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"
Sep 10 16:34:31.116285 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:31.116250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739e13de-b90f-434d-805d-e75b1e6c9b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks\" (UID: \"739e13de-b90f-434d-805d-e75b1e6c9b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"
Sep 10 16:34:31.116443 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:31.116320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c4wf\" (UniqueName: \"kubernetes.io/projected/739e13de-b90f-434d-805d-e75b1e6c9b2e-kube-api-access-2c4wf\") pod \"payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks\" (UID: \"739e13de-b90f-434d-805d-e75b1e6c9b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"
Sep 10 16:34:31.116914 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:31.116894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739e13de-b90f-434d-805d-e75b1e6c9b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks\" (UID: \"739e13de-b90f-434d-805d-e75b1e6c9b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"
Sep 10 16:34:31.124312 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:31.124288    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c4wf\" (UniqueName: \"kubernetes.io/projected/739e13de-b90f-434d-805d-e75b1e6c9b2e-kube-api-access-2c4wf\") pod \"payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks\" (UID: \"739e13de-b90f-434d-805d-e75b1e6c9b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"
Sep 10 16:34:31.195297 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:31.195240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"
Sep 10 16:34:31.463000 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:31.462969    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"]
Sep 10 16:34:31.464158 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:34:31.464134    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739e13de_b90f_434d_805d_e75b1e6c9b2e.slice/crio-238b110ab520285fbff225011e7a108bd4f56723ee1fb94be3010090dbd5ceaa WatchSource:0}: Error finding container 238b110ab520285fbff225011e7a108bd4f56723ee1fb94be3010090dbd5ceaa: Status 404 returned error can't find the container with id 238b110ab520285fbff225011e7a108bd4f56723ee1fb94be3010090dbd5ceaa
Sep 10 16:34:31.563068 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:31.563042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks" event={"ID":"739e13de-b90f-434d-805d-e75b1e6c9b2e","Type":"ContainerStarted","Data":"42b4e8d4c88ce5c79017c437f5aff1620bdd86e621cb19d8d2d8a9f97185e202"}
Sep 10 16:34:31.563187 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:31.563091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks" event={"ID":"739e13de-b90f-434d-805d-e75b1e6c9b2e","Type":"ContainerStarted","Data":"238b110ab520285fbff225011e7a108bd4f56723ee1fb94be3010090dbd5ceaa"}
Sep 10 16:34:31.563187 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:31.563131    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"
Sep 10 16:34:31.579877 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:31.579832    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks" podStartSLOduration=1.5798193170000001 podStartE2EDuration="1.579819317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:34:31.579170821 +0000 UTC m=+1029.780213236" watchObservedRunningTime="2026-09-10 16:34:31.579819317 +0000 UTC m=+1029.780861743"
Sep 10 16:34:42.569359 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:42.569323    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"
Sep 10 16:34:54.687601 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.687567    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"]
Sep 10 16:34:54.690756 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.690738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"
Sep 10 16:34:54.693102 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.693082    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-bd1ece-dockercfg-262nj\""
Sep 10 16:34:54.693213 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.693083    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-bd1ece\""
Sep 10 16:34:54.705833 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.705813    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"]
Sep 10 16:34:54.727786 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.727760    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"]
Sep 10 16:34:54.730906 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.730891    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"
Sep 10 16:34:54.739105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.739086    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"]
Sep 10 16:34:54.790644 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.790612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc7sg\" (UniqueName: \"kubernetes.io/projected/f87220fa-3685-4b19-ba33-538f662d6871-kube-api-access-vc7sg\") pod \"payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2\" (UID: \"f87220fa-3685-4b19-ba33-538f662d6871\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"
Sep 10 16:34:54.790775 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.790681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71e1269-02e8-45e7-83be-0c1706594a47-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7\" (UID: \"f71e1269-02e8-45e7-83be-0c1706594a47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"
Sep 10 16:34:54.790837 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.790791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87220fa-3685-4b19-ba33-538f662d6871-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2\" (UID: \"f87220fa-3685-4b19-ba33-538f662d6871\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"
Sep 10 16:34:54.790890 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.790847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhdgg\" (UniqueName: \"kubernetes.io/projected/f71e1269-02e8-45e7-83be-0c1706594a47-kube-api-access-nhdgg\") pod \"payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7\" (UID: \"f71e1269-02e8-45e7-83be-0c1706594a47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"
Sep 10 16:34:54.891495 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.891464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87220fa-3685-4b19-ba33-538f662d6871-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2\" (UID: \"f87220fa-3685-4b19-ba33-538f662d6871\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"
Sep 10 16:34:54.891639 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.891507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhdgg\" (UniqueName: \"kubernetes.io/projected/f71e1269-02e8-45e7-83be-0c1706594a47-kube-api-access-nhdgg\") pod \"payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7\" (UID: \"f71e1269-02e8-45e7-83be-0c1706594a47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"
Sep 10 16:34:54.891639 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.891622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc7sg\" (UniqueName: \"kubernetes.io/projected/f87220fa-3685-4b19-ba33-538f662d6871-kube-api-access-vc7sg\") pod \"payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2\" (UID: \"f87220fa-3685-4b19-ba33-538f662d6871\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"
Sep 10 16:34:54.891735 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.891653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71e1269-02e8-45e7-83be-0c1706594a47-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7\" (UID: \"f71e1269-02e8-45e7-83be-0c1706594a47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"
Sep 10 16:34:54.892274 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.892254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71e1269-02e8-45e7-83be-0c1706594a47-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7\" (UID: \"f71e1269-02e8-45e7-83be-0c1706594a47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"
Sep 10 16:34:54.892325 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.892275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87220fa-3685-4b19-ba33-538f662d6871-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2\" (UID: \"f87220fa-3685-4b19-ba33-538f662d6871\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"
Sep 10 16:34:54.899403 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.899378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc7sg\" (UniqueName: \"kubernetes.io/projected/f87220fa-3685-4b19-ba33-538f662d6871-kube-api-access-vc7sg\") pod \"payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2\" (UID: \"f87220fa-3685-4b19-ba33-538f662d6871\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"
Sep 10 16:34:54.899692 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:54.899677    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhdgg\" (UniqueName: \"kubernetes.io/projected/f71e1269-02e8-45e7-83be-0c1706594a47-kube-api-access-nhdgg\") pod \"payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7\" (UID: \"f71e1269-02e8-45e7-83be-0c1706594a47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"
Sep 10 16:34:55.000950 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:55.000900    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"
Sep 10 16:34:55.040980 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:55.040947    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"
Sep 10 16:34:55.269004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:55.268982    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"]
Sep 10 16:34:55.270865 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:34:55.270839    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87220fa_3685_4b19_ba33_538f662d6871.slice/crio-77c284fed0afb3110116c921541528b95a274037a0be3377c9821c0c1dca5f1e WatchSource:0}: Error finding container 77c284fed0afb3110116c921541528b95a274037a0be3377c9821c0c1dca5f1e: Status 404 returned error can't find the container with id 77c284fed0afb3110116c921541528b95a274037a0be3377c9821c0c1dca5f1e
Sep 10 16:34:55.320486 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:55.320416    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"]
Sep 10 16:34:55.323463 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:34:55.323441    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71e1269_02e8_45e7_83be_0c1706594a47.slice/crio-97b20e5cc81dd2865b7b2e393408a68650558840cd2ae858831cda266fd73ffc WatchSource:0}: Error finding container 97b20e5cc81dd2865b7b2e393408a68650558840cd2ae858831cda266fd73ffc: Status 404 returned error can't find the container with id 97b20e5cc81dd2865b7b2e393408a68650558840cd2ae858831cda266fd73ffc
Sep 10 16:34:55.668752 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:55.668669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7" event={"ID":"f71e1269-02e8-45e7-83be-0c1706594a47","Type":"ContainerStarted","Data":"85be887bf503b577f80a27ba86f37230f9ae9d67b51942a84a4bf07cf3987cf0"}
Sep 10 16:34:55.668752 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:55.668727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7" event={"ID":"f71e1269-02e8-45e7-83be-0c1706594a47","Type":"ContainerStarted","Data":"97b20e5cc81dd2865b7b2e393408a68650558840cd2ae858831cda266fd73ffc"}
Sep 10 16:34:55.668954 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:55.668756    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"
Sep 10 16:34:55.670260 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:55.670230    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2" event={"ID":"f87220fa-3685-4b19-ba33-538f662d6871","Type":"ContainerStarted","Data":"494eaa647581b088d44669e4ec96112ae21beace8d9c681fe5ab80fd5d7a824c"}
Sep 10 16:34:55.670372 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:55.670282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2" event={"ID":"f87220fa-3685-4b19-ba33-538f662d6871","Type":"ContainerStarted","Data":"77c284fed0afb3110116c921541528b95a274037a0be3377c9821c0c1dca5f1e"}
Sep 10 16:34:55.670372 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:55.670309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"
Sep 10 16:34:55.686851 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:55.686810    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7" podStartSLOduration=1.686796256 podStartE2EDuration="1.686796256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:34:55.685911963 +0000 UTC m=+1053.886954373" watchObservedRunningTime="2026-09-10 16:34:55.686796256 +0000 UTC m=+1053.887838664"
Sep 10 16:34:55.701281 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:55.701242    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2" podStartSLOduration=1.701228486 podStartE2EDuration="1.701228486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:34:55.700576812 +0000 UTC m=+1053.901619224" watchObservedRunningTime="2026-09-10 16:34:55.701228486 +0000 UTC m=+1053.902270896"
Sep 10 16:34:59.528178 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.528144    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"]
Sep 10 16:34:59.531520 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.531503    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"
Sep 10 16:34:59.533801 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.533772    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-3ccbd4\""
Sep 10 16:34:59.533967 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.533910    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-3ccbd4-dockercfg-cx6fm\""
Sep 10 16:34:59.542368 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.542347    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"]
Sep 10 16:34:59.571060 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.571040    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"]
Sep 10 16:34:59.574136 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.574122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"
Sep 10 16:34:59.582398 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.582376    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"]
Sep 10 16:34:59.630754 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.630731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5sc\" (UniqueName: \"kubernetes.io/projected/2832d858-e5bd-45a5-9f6c-3b509a8a454c-kube-api-access-dq5sc\") pod \"payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh\" (UID: \"2832d858-e5bd-45a5-9f6c-3b509a8a454c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"
Sep 10 16:34:59.630862 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.630762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf56v\" (UniqueName: \"kubernetes.io/projected/a696f483-bbba-4458-a279-9b0aa2cf5733-kube-api-access-sf56v\") pod \"payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj\" (UID: \"a696f483-bbba-4458-a279-9b0aa2cf5733\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"
Sep 10 16:34:59.630862 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.630783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2832d858-e5bd-45a5-9f6c-3b509a8a454c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh\" (UID: \"2832d858-e5bd-45a5-9f6c-3b509a8a454c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"
Sep 10 16:34:59.630862 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.630799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a696f483-bbba-4458-a279-9b0aa2cf5733-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj\" (UID: \"a696f483-bbba-4458-a279-9b0aa2cf5733\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"
Sep 10 16:34:59.731278 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.731249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5sc\" (UniqueName: \"kubernetes.io/projected/2832d858-e5bd-45a5-9f6c-3b509a8a454c-kube-api-access-dq5sc\") pod \"payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh\" (UID: \"2832d858-e5bd-45a5-9f6c-3b509a8a454c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"
Sep 10 16:34:59.731400 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.731282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf56v\" (UniqueName: \"kubernetes.io/projected/a696f483-bbba-4458-a279-9b0aa2cf5733-kube-api-access-sf56v\") pod \"payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj\" (UID: \"a696f483-bbba-4458-a279-9b0aa2cf5733\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"
Sep 10 16:34:59.731400 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.731303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2832d858-e5bd-45a5-9f6c-3b509a8a454c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh\" (UID: \"2832d858-e5bd-45a5-9f6c-3b509a8a454c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"
Sep 10 16:34:59.731400 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.731324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a696f483-bbba-4458-a279-9b0aa2cf5733-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj\" (UID: \"a696f483-bbba-4458-a279-9b0aa2cf5733\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"
Sep 10 16:34:59.731974 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.731954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2832d858-e5bd-45a5-9f6c-3b509a8a454c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh\" (UID: \"2832d858-e5bd-45a5-9f6c-3b509a8a454c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"
Sep 10 16:34:59.732026 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.731965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a696f483-bbba-4458-a279-9b0aa2cf5733-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj\" (UID: \"a696f483-bbba-4458-a279-9b0aa2cf5733\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"
Sep 10 16:34:59.738957 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.738932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf56v\" (UniqueName: \"kubernetes.io/projected/a696f483-bbba-4458-a279-9b0aa2cf5733-kube-api-access-sf56v\") pod \"payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj\" (UID: \"a696f483-bbba-4458-a279-9b0aa2cf5733\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"
Sep 10 16:34:59.738957 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.738947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5sc\" (UniqueName: \"kubernetes.io/projected/2832d858-e5bd-45a5-9f6c-3b509a8a454c-kube-api-access-dq5sc\") pod \"payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh\" (UID: \"2832d858-e5bd-45a5-9f6c-3b509a8a454c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"
Sep 10 16:34:59.842336 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.842278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"
Sep 10 16:34:59.883559 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:34:59.883513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"
Sep 10 16:35:00.114356 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:00.114290    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"]
Sep 10 16:35:00.116521 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:35:00.116494    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda696f483_bbba_4458_a279_9b0aa2cf5733.slice/crio-ae441a79a966f56795bcefed6f41646f8fcf743b507084d6416dfa9ab1e98e9e WatchSource:0}: Error finding container ae441a79a966f56795bcefed6f41646f8fcf743b507084d6416dfa9ab1e98e9e: Status 404 returned error can't find the container with id ae441a79a966f56795bcefed6f41646f8fcf743b507084d6416dfa9ab1e98e9e
Sep 10 16:35:00.167730 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:00.167708    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"]
Sep 10 16:35:00.169485 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:35:00.169459    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2832d858_e5bd_45a5_9f6c_3b509a8a454c.slice/crio-483bb4ca3e776cbd73166f73fbc796b71f2217ffe49ade6e8b6d37b008af47ec WatchSource:0}: Error finding container 483bb4ca3e776cbd73166f73fbc796b71f2217ffe49ade6e8b6d37b008af47ec: Status 404 returned error can't find the container with id 483bb4ca3e776cbd73166f73fbc796b71f2217ffe49ade6e8b6d37b008af47ec
Sep 10 16:35:00.700756 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:00.700712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh" event={"ID":"2832d858-e5bd-45a5-9f6c-3b509a8a454c","Type":"ContainerStarted","Data":"a0e4083467ebc4f389fbb4fa31cd86f4117763cadcb9336a8aa3fb3adcd39c4a"}
Sep 10 16:35:00.701211 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:00.700777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh" event={"ID":"2832d858-e5bd-45a5-9f6c-3b509a8a454c","Type":"ContainerStarted","Data":"483bb4ca3e776cbd73166f73fbc796b71f2217ffe49ade6e8b6d37b008af47ec"}
Sep 10 16:35:00.701211 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:00.700808    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"
Sep 10 16:35:00.702449 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:00.702425    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj" event={"ID":"a696f483-bbba-4458-a279-9b0aa2cf5733","Type":"ContainerStarted","Data":"62cbec8170d66767ea66beed45c2d7572339f2e69a431f4be8abd0754c8d2172"}
Sep 10 16:35:00.702600 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:00.702464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj" event={"ID":"a696f483-bbba-4458-a279-9b0aa2cf5733","Type":"ContainerStarted","Data":"ae441a79a966f56795bcefed6f41646f8fcf743b507084d6416dfa9ab1e98e9e"}
Sep 10 16:35:00.702600 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:00.702565    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"
Sep 10 16:35:00.722384 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:00.722342    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh" podStartSLOduration=1.722324933 podStartE2EDuration="1.722324933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:35:00.722157966 +0000 UTC m=+1058.923200380" watchObservedRunningTime="2026-09-10 16:35:00.722324933 +0000 UTC m=+1058.923367343"
Sep 10 16:35:00.743474 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:00.743433    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj" podStartSLOduration=1.743420919 podStartE2EDuration="1.743420919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:35:00.741942155 +0000 UTC m=+1058.942984567" watchObservedRunningTime="2026-09-10 16:35:00.743420919 +0000 UTC m=+1058.944463329"
Sep 10 16:35:06.676477 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:06.676445    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"
Sep 10 16:35:06.677029 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:06.676508    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"
Sep 10 16:35:11.708604 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:11.708574    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"
Sep 10 16:35:11.708972 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:11.708649    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"
Sep 10 16:35:30.350520 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.350448    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"]
Sep 10 16:35:30.354816 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.354798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.359097 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.359075    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1925d8\"/\"openshift-service-ca.crt\""
Sep 10 16:35:30.359097 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.359086    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1925d8\"/\"default-dockercfg-56cxm\""
Sep 10 16:35:30.359275 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.359085    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1925d8\"/\"auth-test-model-1925d8-kserve-self-signed-certs\""
Sep 10 16:35:30.359275 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.359158    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1925d8\"/\"kube-root-ca.crt\""
Sep 10 16:35:30.367621 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.367600    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"]
Sep 10 16:35:30.468249 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.468222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9lqq\" (UniqueName: \"kubernetes.io/projected/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-kube-api-access-d9lqq\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.468362 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.468257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-model-cache\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.468362 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.468284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-home\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.468442 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.468366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-tmp-dir\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.468442 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.468398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-dshm\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.468442 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.468418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-tls-certs\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.569622 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.569594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-model-cache\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.569744 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.569645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-home\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.569744 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.569682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-tmp-dir\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.569744 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.569701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-dshm\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.569744 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.569719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-tls-certs\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.569906 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.569761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9lqq\" (UniqueName: \"kubernetes.io/projected/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-kube-api-access-d9lqq\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.570037 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.570015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-model-cache\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.570082 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.570039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-home\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.570127 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.570086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-tmp-dir\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.571970 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.571942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-dshm\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.572084 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.572005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-tls-certs\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.579177 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.579147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9lqq\" (UniqueName: \"kubernetes.io/projected/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-kube-api-access-d9lqq\") pod \"auth-test-model-1925d8-kserve-b78bf9b95-v6hhd\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.664311 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.664258    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:30.935637 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:30.935610    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"]
Sep 10 16:35:30.937635 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:35:30.937607    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a5fe9bb_0f88_4898_a90d_ecd72bbadf10.slice/crio-5cc64558c72664b1a3fc166325162d5c6da1e395f5c5ed0ab486688c420bb81f WatchSource:0}: Error finding container 5cc64558c72664b1a3fc166325162d5c6da1e395f5c5ed0ab486688c420bb81f: Status 404 returned error can't find the container with id 5cc64558c72664b1a3fc166325162d5c6da1e395f5c5ed0ab486688c420bb81f
Sep 10 16:35:31.841715 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:31.841677    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd" event={"ID":"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10","Type":"ContainerStarted","Data":"48e02200f02f3daa97cb1e00383678f617b684910e71f5bbf6bd230a63e44738"}
Sep 10 16:35:31.841715 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:31.841719    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd" event={"ID":"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10","Type":"ContainerStarted","Data":"5cc64558c72664b1a3fc166325162d5c6da1e395f5c5ed0ab486688c420bb81f"}
Sep 10 16:35:31.862439 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:31.862391    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd" podStartSLOduration=1.862376653 podStartE2EDuration="1.862376653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:35:31.861572195 +0000 UTC m=+1090.062614607" watchObservedRunningTime="2026-09-10 16:35:31.862376653 +0000 UTC m=+1090.063419109"
Sep 10 16:35:40.664399 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:40.664355    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:40.664920 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:40.664409    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:40.677062 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:40.677025    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:35:40.896021 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:35:40.895986    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:36:01.314158 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.314122    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"]
Sep 10 16:36:01.317485 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.317468    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"
Sep 10 16:36:01.319807 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.319784    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8438b933\""
Sep 10 16:36:01.335866 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.335847    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"]
Sep 10 16:36:01.416391 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.416367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f3132-3701-4de4-ad0c-962402792a97-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr\" (UID: \"bf7f3132-3701-4de4-ad0c-962402792a97\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"
Sep 10 16:36:01.416494 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.416402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbpdn\" (UniqueName: \"kubernetes.io/projected/bf7f3132-3701-4de4-ad0c-962402792a97-kube-api-access-dbpdn\") pod \"payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr\" (UID: \"bf7f3132-3701-4de4-ad0c-962402792a97\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"
Sep 10 16:36:01.517856 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.517828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f3132-3701-4de4-ad0c-962402792a97-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr\" (UID: \"bf7f3132-3701-4de4-ad0c-962402792a97\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"
Sep 10 16:36:01.518004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.517877    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbpdn\" (UniqueName: \"kubernetes.io/projected/bf7f3132-3701-4de4-ad0c-962402792a97-kube-api-access-dbpdn\") pod \"payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr\" (UID: \"bf7f3132-3701-4de4-ad0c-962402792a97\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"
Sep 10 16:36:01.518469 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.518449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f3132-3701-4de4-ad0c-962402792a97-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr\" (UID: \"bf7f3132-3701-4de4-ad0c-962402792a97\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"
Sep 10 16:36:01.525926 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.525903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbpdn\" (UniqueName: \"kubernetes.io/projected/bf7f3132-3701-4de4-ad0c-962402792a97-kube-api-access-dbpdn\") pod \"payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr\" (UID: \"bf7f3132-3701-4de4-ad0c-962402792a97\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"
Sep 10 16:36:01.627770 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.627697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"
Sep 10 16:36:01.898240 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.898214    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"]
Sep 10 16:36:01.900429 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:36:01.900403    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7f3132_3701_4de4_ad0c_962402792a97.slice/crio-0e6abca3c40ce843345b069c2039ba2205bd9c2958b6fc566ca7d78c06aae7f5 WatchSource:0}: Error finding container 0e6abca3c40ce843345b069c2039ba2205bd9c2958b6fc566ca7d78c06aae7f5: Status 404 returned error can't find the container with id 0e6abca3c40ce843345b069c2039ba2205bd9c2958b6fc566ca7d78c06aae7f5
Sep 10 16:36:01.976729 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.976700    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr" event={"ID":"bf7f3132-3701-4de4-ad0c-962402792a97","Type":"ContainerStarted","Data":"c73f1310aea01c9556556dc917a10977daad1a64242c7e57cda6c75c444d6691"}
Sep 10 16:36:01.976841 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.976753    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr" event={"ID":"bf7f3132-3701-4de4-ad0c-962402792a97","Type":"ContainerStarted","Data":"0e6abca3c40ce843345b069c2039ba2205bd9c2958b6fc566ca7d78c06aae7f5"}
Sep 10 16:36:01.976841 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.976774    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"
Sep 10 16:36:01.998636 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:01.998592    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr" podStartSLOduration=0.998576784 podStartE2EDuration="998.576784ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:36:01.997848498 +0000 UTC m=+1120.198890910" watchObservedRunningTime="2026-09-10 16:36:01.998576784 +0000 UTC m=+1120.199619194"
Sep 10 16:36:12.983133 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:12.983103    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"
Sep 10 16:36:22.337374 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.337344    2580 scope.go:117] "RemoveContainer" containerID="77441092c5a25b5e615d1bc18c96ef154fbc5ee47b279c1d70d7262959aad73a"
Sep 10 16:36:22.345080 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.345059    2580 scope.go:117] "RemoveContainer" containerID="6a7a87b5149b05c8b64ecc1d40e621af1fd1e0e93775a6f85c8f23402fc30216"
Sep 10 16:36:22.449347 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.449318    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"]
Sep 10 16:36:22.453913 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.453895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.456114 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.456090    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ccbd4\"/\"kube-root-ca.crt\""
Sep 10 16:36:22.456793 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.456772    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ccbd4\"/\"auth-test-model-3ccbd4-kserve-self-signed-certs\""
Sep 10 16:36:22.456793 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.456777    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ccbd4\"/\"default-dockercfg-lpb8g\""
Sep 10 16:36:22.456937 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.456815    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ccbd4\"/\"openshift-service-ca.crt\""
Sep 10 16:36:22.461823 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.461751    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"]
Sep 10 16:36:22.588686 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.588596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74fzb\" (UniqueName: \"kubernetes.io/projected/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-kube-api-access-74fzb\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.588686 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.588662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-tmp-dir\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.588896 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.588734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-home\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.588896 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.588760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-dshm\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.588896 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.588792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-tls-certs\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.589007 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.588894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-model-cache\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.689324 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.689280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-home\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.689324 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.689324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-dshm\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.689582 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.689345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-tls-certs\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.689582 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.689383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-model-cache\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.689582 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.689408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74fzb\" (UniqueName: \"kubernetes.io/projected/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-kube-api-access-74fzb\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.689582 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.689455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-tmp-dir\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.689806 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.689773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-home\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.689850 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.689821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-model-cache\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.689895 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.689874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-tmp-dir\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.691739 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.691697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-dshm\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.691971 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.691949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-tls-certs\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.697657 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.697639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74fzb\" (UniqueName: \"kubernetes.io/projected/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-kube-api-access-74fzb\") pod \"auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:22.764360 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:22.764325    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:23.038173 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:23.038144    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"]
Sep 10 16:36:23.041498 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:36:23.041472    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod334e9ea0_f1fe_4bed_a504_00ba1c77ec5d.slice/crio-fad111a8a8ee97cc1f8116fa0f15e5acaadab83dac420dfd740a2fac461d13cf WatchSource:0}: Error finding container fad111a8a8ee97cc1f8116fa0f15e5acaadab83dac420dfd740a2fac461d13cf: Status 404 returned error can't find the container with id fad111a8a8ee97cc1f8116fa0f15e5acaadab83dac420dfd740a2fac461d13cf
Sep 10 16:36:23.079051 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:23.079025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw" event={"ID":"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d","Type":"ContainerStarted","Data":"fad111a8a8ee97cc1f8116fa0f15e5acaadab83dac420dfd740a2fac461d13cf"}
Sep 10 16:36:24.084955 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:24.084920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw" event={"ID":"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d","Type":"ContainerStarted","Data":"368a3f139b9ca903d76047184c97551374ae519e8359176cc67de0ce0b3e406a"}
Sep 10 16:36:24.113431 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:24.113384    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw" podStartSLOduration=2.113364597 podStartE2EDuration="2.113364597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:36:24.110653852 +0000 UTC m=+1142.311696262" watchObservedRunningTime="2026-09-10 16:36:24.113364597 +0000 UTC m=+1142.314407009"
Sep 10 16:36:32.764498 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:32.764465    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:32.765002 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:32.764515    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:32.777004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:32.776983    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:33.137199 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:33.137121    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:36:43.459648 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.459616    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"]
Sep 10 16:36:43.463258 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.463240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.466422 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.466398    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8438b933\"/\"default-dockercfg-wsjm7\""
Sep 10 16:36:43.466422 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.466412    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8438b933\"/\"e2e-lifecycle-model-8438b933-kserve-self-signed-certs\""
Sep 10 16:36:43.466617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.466398    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8438b933\"/\"kube-root-ca.crt\""
Sep 10 16:36:43.466617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.466400    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8438b933\"/\"openshift-service-ca.crt\""
Sep 10 16:36:43.472399 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.472248    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"]
Sep 10 16:36:43.653950 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.653917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-dshm\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.653950 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.653951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqb47\" (UniqueName: \"kubernetes.io/projected/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-kube-api-access-sqb47\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.654275 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.653975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-tmp-dir\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.654275 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.654003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-tls-certs\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.654275 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.654115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-model-cache\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.654275 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.654217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-home\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.755257 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.755196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-home\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.755257 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.755235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-dshm\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.755257 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.755254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqb47\" (UniqueName: \"kubernetes.io/projected/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-kube-api-access-sqb47\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.755469 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.755273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-tmp-dir\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.755469 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.755301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-tls-certs\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.755469 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.755355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-model-cache\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.755653 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.755603    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-home\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.755795 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.755656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-tmp-dir\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.755795 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.755759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-model-cache\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.757484 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.757464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-dshm\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.757661 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.757642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-tls-certs\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.764186 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.764167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqb47\" (UniqueName: \"kubernetes.io/projected/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-kube-api-access-sqb47\") pod \"e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") " pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:43.775032 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:43.775010    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:44.047332 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:44.047301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"]
Sep 10 16:36:44.050291 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:36:44.050263    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod336b9f6c_bba4_4c3b_92f5_42428ab6ae7c.slice/crio-a797481ed2aafaa772d7ba17ce7bfb0d04a4e3ebb26e4988aba830f873964d88 WatchSource:0}: Error finding container a797481ed2aafaa772d7ba17ce7bfb0d04a4e3ebb26e4988aba830f873964d88: Status 404 returned error can't find the container with id a797481ed2aafaa772d7ba17ce7bfb0d04a4e3ebb26e4988aba830f873964d88
Sep 10 16:36:44.176181 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:44.176149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq" event={"ID":"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c","Type":"ContainerStarted","Data":"4886f9e1567f8ded55a2710df4db244e079d9603442022fb5b4d32d0d2f4a65a"}
Sep 10 16:36:44.176314 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:44.176198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq" event={"ID":"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c","Type":"ContainerStarted","Data":"a797481ed2aafaa772d7ba17ce7bfb0d04a4e3ebb26e4988aba830f873964d88"}
Sep 10 16:36:44.200386 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:44.200337    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq" podStartSLOduration=1.200322028 podStartE2EDuration="1.200322028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:36:44.19928311 +0000 UTC m=+1162.400325521" watchObservedRunningTime="2026-09-10 16:36:44.200322028 +0000 UTC m=+1162.401364439"
Sep 10 16:36:53.775503 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:53.775368    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:53.775503 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:53.775475    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:53.789070 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:53.789037    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:36:54.234746 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:36:54.234669    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:37:22.528068 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:22.528037    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"]
Sep 10 16:37:22.528665 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:22.528331    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr" podUID="bf7f3132-3701-4de4-ad0c-962402792a97" containerName="payload-pre-processing" containerID="cri-o://c73f1310aea01c9556556dc917a10977daad1a64242c7e57cda6c75c444d6691" gracePeriod=30
Sep 10 16:37:22.770237 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:22.770212    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"
Sep 10 16:37:22.861242 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:22.861173    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbpdn\" (UniqueName: \"kubernetes.io/projected/bf7f3132-3701-4de4-ad0c-962402792a97-kube-api-access-dbpdn\") pod \"bf7f3132-3701-4de4-ad0c-962402792a97\" (UID: \"bf7f3132-3701-4de4-ad0c-962402792a97\") "
Sep 10 16:37:22.861242 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:22.861235    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f3132-3701-4de4-ad0c-962402792a97-plugins-config-volume\") pod \"bf7f3132-3701-4de4-ad0c-962402792a97\" (UID: \"bf7f3132-3701-4de4-ad0c-962402792a97\") "
Sep 10 16:37:22.861606 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:22.861583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7f3132-3701-4de4-ad0c-962402792a97-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf7f3132-3701-4de4-ad0c-962402792a97" (UID: "bf7f3132-3701-4de4-ad0c-962402792a97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:37:22.863183 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:22.863159    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7f3132-3701-4de4-ad0c-962402792a97-kube-api-access-dbpdn" (OuterVolumeSpecName: "kube-api-access-dbpdn") pod "bf7f3132-3701-4de4-ad0c-962402792a97" (UID: "bf7f3132-3701-4de4-ad0c-962402792a97"). InnerVolumeSpecName "kube-api-access-dbpdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:37:22.961770 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:22.961747    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbpdn\" (UniqueName: \"kubernetes.io/projected/bf7f3132-3701-4de4-ad0c-962402792a97-kube-api-access-dbpdn\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:37:22.961770 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:22.961767    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f3132-3701-4de4-ad0c-962402792a97-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:37:23.353613 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:23.353578    2580 generic.go:356] "Generic (PLEG): container finished" podID="bf7f3132-3701-4de4-ad0c-962402792a97" containerID="c73f1310aea01c9556556dc917a10977daad1a64242c7e57cda6c75c444d6691" exitCode=0
Sep 10 16:37:23.353805 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:23.353624    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"
Sep 10 16:37:23.353805 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:23.353664    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr" event={"ID":"bf7f3132-3701-4de4-ad0c-962402792a97","Type":"ContainerDied","Data":"c73f1310aea01c9556556dc917a10977daad1a64242c7e57cda6c75c444d6691"}
Sep 10 16:37:23.353805 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:23.353722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr" event={"ID":"bf7f3132-3701-4de4-ad0c-962402792a97","Type":"ContainerDied","Data":"0e6abca3c40ce843345b069c2039ba2205bd9c2958b6fc566ca7d78c06aae7f5"}
Sep 10 16:37:23.353805 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:23.353753    2580 scope.go:117] "RemoveContainer" containerID="c73f1310aea01c9556556dc917a10977daad1a64242c7e57cda6c75c444d6691"
Sep 10 16:37:23.362653 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:23.362636    2580 scope.go:117] "RemoveContainer" containerID="c73f1310aea01c9556556dc917a10977daad1a64242c7e57cda6c75c444d6691"
Sep 10 16:37:23.362901 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:37:23.362881    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c73f1310aea01c9556556dc917a10977daad1a64242c7e57cda6c75c444d6691\": container with ID starting with c73f1310aea01c9556556dc917a10977daad1a64242c7e57cda6c75c444d6691 not found: ID does not exist" containerID="c73f1310aea01c9556556dc917a10977daad1a64242c7e57cda6c75c444d6691"
Sep 10 16:37:23.363038 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:23.362906    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c73f1310aea01c9556556dc917a10977daad1a64242c7e57cda6c75c444d6691"} err="failed to get container status \"c73f1310aea01c9556556dc917a10977daad1a64242c7e57cda6c75c444d6691\": rpc error: code = NotFound desc = could not find container \"c73f1310aea01c9556556dc917a10977daad1a64242c7e57cda6c75c444d6691\": container with ID starting with c73f1310aea01c9556556dc917a10977daad1a64242c7e57cda6c75c444d6691 not found: ID does not exist"
Sep 10 16:37:23.381828 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:23.381806    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"]
Sep 10 16:37:23.387689 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:23.387668    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8438b933-679f599db-v4tzr"]
Sep 10 16:37:24.255016 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:24.254979    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf7f3132-3701-4de4-ad0c-962402792a97" path="/var/lib/kubelet/pods/bf7f3132-3701-4de4-ad0c-962402792a97/volumes"
Sep 10 16:37:30.787166 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:30.787127    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"]
Sep 10 16:37:45.806550 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:45.806477    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq" podUID="336b9f6c-bba4-4c3b-92f5-42428ab6ae7c" containerName="main" containerID="cri-o://4886f9e1567f8ded55a2710df4db244e079d9603442022fb5b4d32d0d2f4a65a" gracePeriod=45
Sep 10 16:37:46.052058 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.052028    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:37:46.152642 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.152566    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-tls-certs\") pod \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") "
Sep 10 16:37:46.152642 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.152639    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-model-cache\") pod \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") "
Sep 10 16:37:46.152863 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.152662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-home\") pod \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") "
Sep 10 16:37:46.152863 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.152692    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-dshm\") pod \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") "
Sep 10 16:37:46.152863 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.152723    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-tmp-dir\") pod \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") "
Sep 10 16:37:46.152863 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.152748    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqb47\" (UniqueName: \"kubernetes.io/projected/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-kube-api-access-sqb47\") pod \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\" (UID: \"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c\") "
Sep 10 16:37:46.153057 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.152930    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "336b9f6c-bba4-4c3b-92f5-42428ab6ae7c" (UID: "336b9f6c-bba4-4c3b-92f5-42428ab6ae7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:37:46.153057 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.152943    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-home" (OuterVolumeSpecName: "home") pod "336b9f6c-bba4-4c3b-92f5-42428ab6ae7c" (UID: "336b9f6c-bba4-4c3b-92f5-42428ab6ae7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:37:46.153057 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.152954    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "336b9f6c-bba4-4c3b-92f5-42428ab6ae7c" (UID: "336b9f6c-bba4-4c3b-92f5-42428ab6ae7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:37:46.154799 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.154770    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-dshm" (OuterVolumeSpecName: "dshm") pod "336b9f6c-bba4-4c3b-92f5-42428ab6ae7c" (UID: "336b9f6c-bba4-4c3b-92f5-42428ab6ae7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:37:46.154927 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.154889    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "336b9f6c-bba4-4c3b-92f5-42428ab6ae7c" (UID: "336b9f6c-bba4-4c3b-92f5-42428ab6ae7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:37:46.154983 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.154922    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-kube-api-access-sqb47" (OuterVolumeSpecName: "kube-api-access-sqb47") pod "336b9f6c-bba4-4c3b-92f5-42428ab6ae7c" (UID: "336b9f6c-bba4-4c3b-92f5-42428ab6ae7c"). InnerVolumeSpecName "kube-api-access-sqb47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:37:46.253428 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.253399    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:37:46.253549 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.253431    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqb47\" (UniqueName: \"kubernetes.io/projected/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-kube-api-access-sqb47\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:37:46.253549 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.253447    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:37:46.253549 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.253460    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:37:46.253549 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.253472    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:37:46.253549 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.253487    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:37:46.459374 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.459295    2580 generic.go:356] "Generic (PLEG): container finished" podID="336b9f6c-bba4-4c3b-92f5-42428ab6ae7c" containerID="4886f9e1567f8ded55a2710df4db244e079d9603442022fb5b4d32d0d2f4a65a" exitCode=0
Sep 10 16:37:46.459374 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.459335    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"
Sep 10 16:37:46.459561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.459392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq" event={"ID":"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c","Type":"ContainerDied","Data":"4886f9e1567f8ded55a2710df4db244e079d9603442022fb5b4d32d0d2f4a65a"}
Sep 10 16:37:46.459561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.459433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq" event={"ID":"336b9f6c-bba4-4c3b-92f5-42428ab6ae7c","Type":"ContainerDied","Data":"a797481ed2aafaa772d7ba17ce7bfb0d04a4e3ebb26e4988aba830f873964d88"}
Sep 10 16:37:46.459561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.459452    2580 scope.go:117] "RemoveContainer" containerID="4886f9e1567f8ded55a2710df4db244e079d9603442022fb5b4d32d0d2f4a65a"
Sep 10 16:37:46.467777 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.467759    2580 scope.go:117] "RemoveContainer" containerID="4886f9e1567f8ded55a2710df4db244e079d9603442022fb5b4d32d0d2f4a65a"
Sep 10 16:37:46.468034 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:37:46.468017    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4886f9e1567f8ded55a2710df4db244e079d9603442022fb5b4d32d0d2f4a65a\": container with ID starting with 4886f9e1567f8ded55a2710df4db244e079d9603442022fb5b4d32d0d2f4a65a not found: ID does not exist" containerID="4886f9e1567f8ded55a2710df4db244e079d9603442022fb5b4d32d0d2f4a65a"
Sep 10 16:37:46.468095 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.468040    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4886f9e1567f8ded55a2710df4db244e079d9603442022fb5b4d32d0d2f4a65a"} err="failed to get container status \"4886f9e1567f8ded55a2710df4db244e079d9603442022fb5b4d32d0d2f4a65a\": rpc error: code = NotFound desc = could not find container \"4886f9e1567f8ded55a2710df4db244e079d9603442022fb5b4d32d0d2f4a65a\": container with ID starting with 4886f9e1567f8ded55a2710df4db244e079d9603442022fb5b4d32d0d2f4a65a not found: ID does not exist"
Sep 10 16:37:46.475171 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.475151    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"]
Sep 10 16:37:46.479152 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:46.479133    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8438b933/e2e-lifecycle-model-8438b933-kserve-75b9d6579f-2qqvq"]
Sep 10 16:37:48.254224 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:37:48.254188    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="336b9f6c-bba4-4c3b-92f5-42428ab6ae7c" path="/var/lib/kubelet/pods/336b9f6c-bba4-4c3b-92f5-42428ab6ae7c/volumes"
Sep 10 16:38:32.625837 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:32.625801    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"]
Sep 10 16:38:33.539550 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:33.539511    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"]
Sep 10 16:38:47.645055 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:47.645012    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd" podUID="5a5fe9bb-0f88-4898-a90d-ecd72bbadf10" containerName="main" containerID="cri-o://48e02200f02f3daa97cb1e00383678f617b684910e71f5bbf6bd230a63e44738" gracePeriod=45
Sep 10 16:38:47.890036 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:47.890013    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:38:48.018047 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.018006    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-model-cache\") pod \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") "
Sep 10 16:38:48.018204 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.018061    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-tmp-dir\") pod \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") "
Sep 10 16:38:48.018204 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.018095    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-home\") pod \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") "
Sep 10 16:38:48.018204 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.018118    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-tls-certs\") pod \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") "
Sep 10 16:38:48.018204 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.018163    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9lqq\" (UniqueName: \"kubernetes.io/projected/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-kube-api-access-d9lqq\") pod \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") "
Sep 10 16:38:48.018374 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.018281    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a5fe9bb-0f88-4898-a90d-ecd72bbadf10" (UID: "5a5fe9bb-0f88-4898-a90d-ecd72bbadf10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:38:48.018374 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.018296    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-dshm\") pod \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\" (UID: \"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10\") "
Sep 10 16:38:48.018444 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.018375    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a5fe9bb-0f88-4898-a90d-ecd72bbadf10" (UID: "5a5fe9bb-0f88-4898-a90d-ecd72bbadf10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:38:48.018444 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.018415    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-home" (OuterVolumeSpecName: "home") pod "5a5fe9bb-0f88-4898-a90d-ecd72bbadf10" (UID: "5a5fe9bb-0f88-4898-a90d-ecd72bbadf10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:38:48.018629 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.018605    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:38:48.018629 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.018630    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:38:48.018791 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.018643    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:38:48.020498 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.020469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a5fe9bb-0f88-4898-a90d-ecd72bbadf10" (UID: "5a5fe9bb-0f88-4898-a90d-ecd72bbadf10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:38:48.020498 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.020486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-dshm" (OuterVolumeSpecName: "dshm") pod "5a5fe9bb-0f88-4898-a90d-ecd72bbadf10" (UID: "5a5fe9bb-0f88-4898-a90d-ecd72bbadf10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:38:48.020649 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.020483    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-kube-api-access-d9lqq" (OuterVolumeSpecName: "kube-api-access-d9lqq") pod "5a5fe9bb-0f88-4898-a90d-ecd72bbadf10" (UID: "5a5fe9bb-0f88-4898-a90d-ecd72bbadf10"). InnerVolumeSpecName "kube-api-access-d9lqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:38:48.120078 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.120048    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:38:48.120078 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.120074    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9lqq\" (UniqueName: \"kubernetes.io/projected/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-kube-api-access-d9lqq\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:38:48.120078 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.120084    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:38:48.502617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.500730    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"]
Sep 10 16:38:48.502617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.501457    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf7f3132-3701-4de4-ad0c-962402792a97" containerName="payload-pre-processing"
Sep 10 16:38:48.502617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.501475    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf7f3132-3701-4de4-ad0c-962402792a97" containerName="payload-pre-processing"
Sep 10 16:38:48.502617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.501513    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="336b9f6c-bba4-4c3b-92f5-42428ab6ae7c" containerName="main"
Sep 10 16:38:48.502617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.501521    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="336b9f6c-bba4-4c3b-92f5-42428ab6ae7c" containerName="main"
Sep 10 16:38:48.502617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.501571    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a5fe9bb-0f88-4898-a90d-ecd72bbadf10" containerName="main"
Sep 10 16:38:48.502617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.501580    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a5fe9bb-0f88-4898-a90d-ecd72bbadf10" containerName="main"
Sep 10 16:38:48.502617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.501728    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="336b9f6c-bba4-4c3b-92f5-42428ab6ae7c" containerName="main"
Sep 10 16:38:48.502617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.501748    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a5fe9bb-0f88-4898-a90d-ecd72bbadf10" containerName="main"
Sep 10 16:38:48.502617 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.501757    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf7f3132-3701-4de4-ad0c-962402792a97" containerName="payload-pre-processing"
Sep 10 16:38:48.505715 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.505677    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.508544 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.508508    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1925d8\"/\"sub-test-model-1925d8-kserve-self-signed-certs\""
Sep 10 16:38:48.520497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.520475    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"]
Sep 10 16:38:48.558508 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.558463    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw" podUID="334e9ea0-f1fe-4bed-a504-00ba1c77ec5d" containerName="main" containerID="cri-o://368a3f139b9ca903d76047184c97551374ae519e8359176cc67de0ce0b3e406a" gracePeriod=45
Sep 10 16:38:48.624372 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.624347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-model-cache\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.624473 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.624384    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-home\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.624473 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.624400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/491dd771-69c6-4282-81b9-d3aa5794620f-tls-certs\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.624473 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.624416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6fpg\" (UniqueName: \"kubernetes.io/projected/491dd771-69c6-4282-81b9-d3aa5794620f-kube-api-access-j6fpg\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.624679 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.624524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-dshm\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.624679 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.624600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-tmp-dir\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.725789 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.725759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-dshm\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.726180 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.725820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-tmp-dir\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.726180 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.725907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-model-cache\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.726180 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.725960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-home\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.726180 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.725986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/491dd771-69c6-4282-81b9-d3aa5794620f-tls-certs\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.726180 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.726010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6fpg\" (UniqueName: \"kubernetes.io/projected/491dd771-69c6-4282-81b9-d3aa5794620f-kube-api-access-j6fpg\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.726458 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.726325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-tmp-dir\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.726458 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.726419    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-home\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.726703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.726678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-model-cache\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.728080 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.728053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-dshm\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.728781 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.728757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/491dd771-69c6-4282-81b9-d3aa5794620f-tls-certs\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.736233 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.736209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6fpg\" (UniqueName: \"kubernetes.io/projected/491dd771-69c6-4282-81b9-d3aa5794620f-kube-api-access-j6fpg\") pod \"sub-test-model-1925d8-kserve-6cf4f647f7-wkszw\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.754383 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.754323    2580 generic.go:356] "Generic (PLEG): container finished" podID="5a5fe9bb-0f88-4898-a90d-ecd72bbadf10" containerID="48e02200f02f3daa97cb1e00383678f617b684910e71f5bbf6bd230a63e44738" exitCode=0
Sep 10 16:38:48.754383 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.754374    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"
Sep 10 16:38:48.754546 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.754387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd" event={"ID":"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10","Type":"ContainerDied","Data":"48e02200f02f3daa97cb1e00383678f617b684910e71f5bbf6bd230a63e44738"}
Sep 10 16:38:48.754546 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.754426    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd" event={"ID":"5a5fe9bb-0f88-4898-a90d-ecd72bbadf10","Type":"ContainerDied","Data":"5cc64558c72664b1a3fc166325162d5c6da1e395f5c5ed0ab486688c420bb81f"}
Sep 10 16:38:48.754546 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.754450    2580 scope.go:117] "RemoveContainer" containerID="48e02200f02f3daa97cb1e00383678f617b684910e71f5bbf6bd230a63e44738"
Sep 10 16:38:48.756305 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.756275    2580 generic.go:356] "Generic (PLEG): container finished" podID="334e9ea0-f1fe-4bed-a504-00ba1c77ec5d" containerID="368a3f139b9ca903d76047184c97551374ae519e8359176cc67de0ce0b3e406a" exitCode=0
Sep 10 16:38:48.756470 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.756370    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw" event={"ID":"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d","Type":"ContainerDied","Data":"368a3f139b9ca903d76047184c97551374ae519e8359176cc67de0ce0b3e406a"}
Sep 10 16:38:48.763347 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.763329    2580 scope.go:117] "RemoveContainer" containerID="48e02200f02f3daa97cb1e00383678f617b684910e71f5bbf6bd230a63e44738"
Sep 10 16:38:48.763634 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:38:48.763611    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e02200f02f3daa97cb1e00383678f617b684910e71f5bbf6bd230a63e44738\": container with ID starting with 48e02200f02f3daa97cb1e00383678f617b684910e71f5bbf6bd230a63e44738 not found: ID does not exist" containerID="48e02200f02f3daa97cb1e00383678f617b684910e71f5bbf6bd230a63e44738"
Sep 10 16:38:48.763674 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.763647    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e02200f02f3daa97cb1e00383678f617b684910e71f5bbf6bd230a63e44738"} err="failed to get container status \"48e02200f02f3daa97cb1e00383678f617b684910e71f5bbf6bd230a63e44738\": rpc error: code = NotFound desc = could not find container \"48e02200f02f3daa97cb1e00383678f617b684910e71f5bbf6bd230a63e44738\": container with ID starting with 48e02200f02f3daa97cb1e00383678f617b684910e71f5bbf6bd230a63e44738 not found: ID does not exist"
Sep 10 16:38:48.769480 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.769459    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"]
Sep 10 16:38:48.773546 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.773515    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/auth-test-model-1925d8-kserve-b78bf9b95-v6hhd"]
Sep 10 16:38:48.807880 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.807863    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:38:48.815009 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.814990    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:48.928252 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.928224    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-dshm\") pod \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") "
Sep 10 16:38:48.928417 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.928268    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-tls-certs\") pod \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") "
Sep 10 16:38:48.928417 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.928303    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-tmp-dir\") pod \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") "
Sep 10 16:38:48.928417 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.928386    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-model-cache\") pod \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") "
Sep 10 16:38:48.928624 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.928422    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74fzb\" (UniqueName: \"kubernetes.io/projected/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-kube-api-access-74fzb\") pod \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") "
Sep 10 16:38:48.928624 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.928476    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-home\") pod \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\" (UID: \"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d\") "
Sep 10 16:38:48.928735 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.928632    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "334e9ea0-f1fe-4bed-a504-00ba1c77ec5d" (UID: "334e9ea0-f1fe-4bed-a504-00ba1c77ec5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:38:48.928735 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.928648    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "334e9ea0-f1fe-4bed-a504-00ba1c77ec5d" (UID: "334e9ea0-f1fe-4bed-a504-00ba1c77ec5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:38:48.928834 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.928756    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-home" (OuterVolumeSpecName: "home") pod "334e9ea0-f1fe-4bed-a504-00ba1c77ec5d" (UID: "334e9ea0-f1fe-4bed-a504-00ba1c77ec5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:38:48.928894 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.928857    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:38:48.928894 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.928875    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:38:48.928894 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.928886    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:38:48.930376 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.930349    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-dshm" (OuterVolumeSpecName: "dshm") pod "334e9ea0-f1fe-4bed-a504-00ba1c77ec5d" (UID: "334e9ea0-f1fe-4bed-a504-00ba1c77ec5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:38:48.930480 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.930394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "334e9ea0-f1fe-4bed-a504-00ba1c77ec5d" (UID: "334e9ea0-f1fe-4bed-a504-00ba1c77ec5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:38:48.930526 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:48.930472    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-kube-api-access-74fzb" (OuterVolumeSpecName: "kube-api-access-74fzb") pod "334e9ea0-f1fe-4bed-a504-00ba1c77ec5d" (UID: "334e9ea0-f1fe-4bed-a504-00ba1c77ec5d"). InnerVolumeSpecName "kube-api-access-74fzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:38:49.029845 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:49.029820    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74fzb\" (UniqueName: \"kubernetes.io/projected/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-kube-api-access-74fzb\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:38:49.029845 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:49.029840    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:38:49.029984 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:49.029850    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:38:49.087833 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:49.087809    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"]
Sep 10 16:38:49.090195 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:38:49.090169    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod491dd771_69c6_4282_81b9_d3aa5794620f.slice/crio-2979b5afe7d3903fdb95b43f7abe2323bc8f8a213b99381da911abe49c49e2b9 WatchSource:0}: Error finding container 2979b5afe7d3903fdb95b43f7abe2323bc8f8a213b99381da911abe49c49e2b9: Status 404 returned error can't find the container with id 2979b5afe7d3903fdb95b43f7abe2323bc8f8a213b99381da911abe49c49e2b9
Sep 10 16:38:49.762632 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:49.762590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw" event={"ID":"491dd771-69c6-4282-81b9-d3aa5794620f","Type":"ContainerStarted","Data":"1af19fe9b8b58e9e92b470efa6e6941a3361934b3418c81dd6a828e3251695e1"}
Sep 10 16:38:49.763037 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:49.762640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw" event={"ID":"491dd771-69c6-4282-81b9-d3aa5794620f","Type":"ContainerStarted","Data":"2979b5afe7d3903fdb95b43f7abe2323bc8f8a213b99381da911abe49c49e2b9"}
Sep 10 16:38:49.764675 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:49.764655    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"
Sep 10 16:38:49.764820 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:49.764681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw" event={"ID":"334e9ea0-f1fe-4bed-a504-00ba1c77ec5d","Type":"ContainerDied","Data":"fad111a8a8ee97cc1f8116fa0f15e5acaadab83dac420dfd740a2fac461d13cf"}
Sep 10 16:38:49.764820 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:49.764727    2580 scope.go:117] "RemoveContainer" containerID="368a3f139b9ca903d76047184c97551374ae519e8359176cc67de0ce0b3e406a"
Sep 10 16:38:49.781027 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:49.780988    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw" podStartSLOduration=1.780962745 podStartE2EDuration="1.780962745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:38:49.779357816 +0000 UTC m=+1287.980400227" watchObservedRunningTime="2026-09-10 16:38:49.780962745 +0000 UTC m=+1287.982005155"
Sep 10 16:38:49.790935 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:49.790916    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"]
Sep 10 16:38:49.796844 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:49.796824    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/auth-test-model-3ccbd4-kserve-684fcb895f-lvmvw"]
Sep 10 16:38:50.255026 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:50.254988    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="334e9ea0-f1fe-4bed-a504-00ba1c77ec5d" path="/var/lib/kubelet/pods/334e9ea0-f1fe-4bed-a504-00ba1c77ec5d/volumes"
Sep 10 16:38:50.255441 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:50.255419    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a5fe9bb-0f88-4898-a90d-ecd72bbadf10" path="/var/lib/kubelet/pods/5a5fe9bb-0f88-4898-a90d-ecd72bbadf10/volumes"
Sep 10 16:38:58.815786 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:58.815748    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:58.816202 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:58.815851    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:58.829271 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:58.829252    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:38:59.821697 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:38:59.821670    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:39:10.759480 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.759449    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"]
Sep 10 16:39:10.760011 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.759810    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334e9ea0-f1fe-4bed-a504-00ba1c77ec5d" containerName="main"
Sep 10 16:39:10.760011 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.759822    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="334e9ea0-f1fe-4bed-a504-00ba1c77ec5d" containerName="main"
Sep 10 16:39:10.760011 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.759882    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="334e9ea0-f1fe-4bed-a504-00ba1c77ec5d" containerName="main"
Sep 10 16:39:10.762728 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.762713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:10.765732 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.765708    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ccbd4\"/\"openshift-service-ca.crt\""
Sep 10 16:39:10.765857 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.765831    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ccbd4\"/\"default-dockercfg-lpb8g\""
Sep 10 16:39:10.766513 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.766495    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ccbd4\"/\"sub-test-model-3ccbd4-kserve-self-signed-certs\""
Sep 10 16:39:10.766625 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.766515    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ccbd4\"/\"kube-root-ca.crt\""
Sep 10 16:39:10.778494 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.778473    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"]
Sep 10 16:39:10.899867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.899835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-tmp-dir\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:10.900026 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.899872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/196a9276-b620-4f30-b77c-5e26b6b81e5a-tls-certs\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:10.900026 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.899993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-home\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:10.900124 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.900024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-dshm\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:10.900124 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.900057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47l9p\" (UniqueName: \"kubernetes.io/projected/196a9276-b620-4f30-b77c-5e26b6b81e5a-kube-api-access-47l9p\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:10.900124 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:10.900115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-model-cache\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:11.001353 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.001314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-tmp-dir\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:11.001353 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.001357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/196a9276-b620-4f30-b77c-5e26b6b81e5a-tls-certs\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:11.001581 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.001439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-home\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:11.001581 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.001469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-dshm\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:11.001581 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.001500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47l9p\" (UniqueName: \"kubernetes.io/projected/196a9276-b620-4f30-b77c-5e26b6b81e5a-kube-api-access-47l9p\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:11.001581 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.001553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-model-cache\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:11.001812 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.001750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-tmp-dir\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:11.001870 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.001826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-home\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:11.001906 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.001885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-model-cache\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:11.003673 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.003654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-dshm\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:11.003852 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.003833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/196a9276-b620-4f30-b77c-5e26b6b81e5a-tls-certs\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:11.010814 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.010761    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47l9p\" (UniqueName: \"kubernetes.io/projected/196a9276-b620-4f30-b77c-5e26b6b81e5a-kube-api-access-47l9p\") pod \"sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:11.071790 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.071767    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:11.345453 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.345281    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"]
Sep 10 16:39:11.348048 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:39:11.348013    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod196a9276_b620_4f30_b77c_5e26b6b81e5a.slice/crio-7ac14c6425c91c70e94523ed4fc0189f668748438fbc656b158e655416d49ea1 WatchSource:0}: Error finding container 7ac14c6425c91c70e94523ed4fc0189f668748438fbc656b158e655416d49ea1: Status 404 returned error can't find the container with id 7ac14c6425c91c70e94523ed4fc0189f668748438fbc656b158e655416d49ea1
Sep 10 16:39:11.865543 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.865502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk" event={"ID":"196a9276-b620-4f30-b77c-5e26b6b81e5a","Type":"ContainerStarted","Data":"bc2fe43f8df8498a2409f3806b0f3eb88ef810c2d49fa025bb14c1497900f8e6"}
Sep 10 16:39:11.865946 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.865573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk" event={"ID":"196a9276-b620-4f30-b77c-5e26b6b81e5a","Type":"ContainerStarted","Data":"7ac14c6425c91c70e94523ed4fc0189f668748438fbc656b158e655416d49ea1"}
Sep 10 16:39:11.884718 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:11.884671    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk" podStartSLOduration=1.884655495 podStartE2EDuration="1.884655495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:39:11.882383293 +0000 UTC m=+1310.083425704" watchObservedRunningTime="2026-09-10 16:39:11.884655495 +0000 UTC m=+1310.085697963"
Sep 10 16:39:21.072131 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:21.072091    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:21.072561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:21.072152    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:21.087698 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:21.087662    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:21.925831 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:21.925803    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:39:41.103513 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.103480    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"]
Sep 10 16:39:41.108129 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.108112    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"
Sep 10 16:39:41.110754 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.110733    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fb3d69\""
Sep 10 16:39:41.117561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.117522    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"]
Sep 10 16:39:41.132155 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.132134    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"]
Sep 10 16:39:41.135379 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.135360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"
Sep 10 16:39:41.139110 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.139089    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fb3d69-dockercfg-b6rv9\""
Sep 10 16:39:41.144419 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.144397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79nh\" (UniqueName: \"kubernetes.io/projected/2477c2fa-f4e8-458c-b5be-09af9f69aa88-kube-api-access-z79nh\") pod \"payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq\" (UID: \"2477c2fa-f4e8-458c-b5be-09af9f69aa88\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"
Sep 10 16:39:41.144510 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.144455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2477c2fa-f4e8-458c-b5be-09af9f69aa88-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq\" (UID: \"2477c2fa-f4e8-458c-b5be-09af9f69aa88\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"
Sep 10 16:39:41.148153 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.148128    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"]
Sep 10 16:39:41.245847 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.245818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2477c2fa-f4e8-458c-b5be-09af9f69aa88-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq\" (UID: \"2477c2fa-f4e8-458c-b5be-09af9f69aa88\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"
Sep 10 16:39:41.245992 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.245855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb20ecaa-57f4-4393-8091-bf8fcfc14937-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng\" (UID: \"cb20ecaa-57f4-4393-8091-bf8fcfc14937\") " pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"
Sep 10 16:39:41.245992 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.245893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwkv\" (UniqueName: \"kubernetes.io/projected/cb20ecaa-57f4-4393-8091-bf8fcfc14937-kube-api-access-xqwkv\") pod \"payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng\" (UID: \"cb20ecaa-57f4-4393-8091-bf8fcfc14937\") " pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"
Sep 10 16:39:41.245992 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.245977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79nh\" (UniqueName: \"kubernetes.io/projected/2477c2fa-f4e8-458c-b5be-09af9f69aa88-kube-api-access-z79nh\") pod \"payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq\" (UID: \"2477c2fa-f4e8-458c-b5be-09af9f69aa88\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"
Sep 10 16:39:41.246491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.246473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2477c2fa-f4e8-458c-b5be-09af9f69aa88-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq\" (UID: \"2477c2fa-f4e8-458c-b5be-09af9f69aa88\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"
Sep 10 16:39:41.254559 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.254520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79nh\" (UniqueName: \"kubernetes.io/projected/2477c2fa-f4e8-458c-b5be-09af9f69aa88-kube-api-access-z79nh\") pod \"payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq\" (UID: \"2477c2fa-f4e8-458c-b5be-09af9f69aa88\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"
Sep 10 16:39:41.346788 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.346758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb20ecaa-57f4-4393-8091-bf8fcfc14937-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng\" (UID: \"cb20ecaa-57f4-4393-8091-bf8fcfc14937\") " pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"
Sep 10 16:39:41.346944 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.346812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwkv\" (UniqueName: \"kubernetes.io/projected/cb20ecaa-57f4-4393-8091-bf8fcfc14937-kube-api-access-xqwkv\") pod \"payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng\" (UID: \"cb20ecaa-57f4-4393-8091-bf8fcfc14937\") " pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"
Sep 10 16:39:41.347493 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.347467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb20ecaa-57f4-4393-8091-bf8fcfc14937-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng\" (UID: \"cb20ecaa-57f4-4393-8091-bf8fcfc14937\") " pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"
Sep 10 16:39:41.355324 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.355278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwkv\" (UniqueName: \"kubernetes.io/projected/cb20ecaa-57f4-4393-8091-bf8fcfc14937-kube-api-access-xqwkv\") pod \"payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng\" (UID: \"cb20ecaa-57f4-4393-8091-bf8fcfc14937\") " pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"
Sep 10 16:39:41.419167 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.419142    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"
Sep 10 16:39:41.445468 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.445442    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"
Sep 10 16:39:41.694028 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.693991    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"]
Sep 10 16:39:41.695326 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:39:41.695300    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2477c2fa_f4e8_458c_b5be_09af9f69aa88.slice/crio-00509f859655bc2e0318b3388f4ec8ae8f6356353eec466addaf28302474e5e5 WatchSource:0}: Error finding container 00509f859655bc2e0318b3388f4ec8ae8f6356353eec466addaf28302474e5e5: Status 404 returned error can't find the container with id 00509f859655bc2e0318b3388f4ec8ae8f6356353eec466addaf28302474e5e5
Sep 10 16:39:41.727509 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:41.727478    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"]
Sep 10 16:39:41.730856 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:39:41.730832    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb20ecaa_57f4_4393_8091_bf8fcfc14937.slice/crio-441adf06ea5a9b024fbc93faa98785fe19d6e53d9d3324e23ac9901bfeeb3092 WatchSource:0}: Error finding container 441adf06ea5a9b024fbc93faa98785fe19d6e53d9d3324e23ac9901bfeeb3092: Status 404 returned error can't find the container with id 441adf06ea5a9b024fbc93faa98785fe19d6e53d9d3324e23ac9901bfeeb3092
Sep 10 16:39:42.017153 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:42.017116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq" event={"ID":"2477c2fa-f4e8-458c-b5be-09af9f69aa88","Type":"ContainerStarted","Data":"043c880dafc8dc67c63d3abd32d8806c7fac037e998d734ec07d87d17e977676"}
Sep 10 16:39:42.017462 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:42.017216    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"
Sep 10 16:39:42.017462 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:42.017242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq" event={"ID":"2477c2fa-f4e8-458c-b5be-09af9f69aa88","Type":"ContainerStarted","Data":"00509f859655bc2e0318b3388f4ec8ae8f6356353eec466addaf28302474e5e5"}
Sep 10 16:39:42.018847 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:42.018817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng" event={"ID":"cb20ecaa-57f4-4393-8091-bf8fcfc14937","Type":"ContainerStarted","Data":"ceb0001b7be091b309e685be56e57f0af50cdb5db82a569d78439a4c1d5ef938"}
Sep 10 16:39:42.018942 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:42.018872    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"
Sep 10 16:39:42.018942 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:42.018895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng" event={"ID":"cb20ecaa-57f4-4393-8091-bf8fcfc14937","Type":"ContainerStarted","Data":"441adf06ea5a9b024fbc93faa98785fe19d6e53d9d3324e23ac9901bfeeb3092"}
Sep 10 16:39:42.041136 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:42.041090    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq" podStartSLOduration=1.041076688 podStartE2EDuration="1.041076688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:39:42.038979587 +0000 UTC m=+1340.240021998" watchObservedRunningTime="2026-09-10 16:39:42.041076688 +0000 UTC m=+1340.242119212"
Sep 10 16:39:42.061732 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:42.061682    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng" podStartSLOduration=1.061664138 podStartE2EDuration="1.061664138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:39:42.060564148 +0000 UTC m=+1340.261606560" watchObservedRunningTime="2026-09-10 16:39:42.061664138 +0000 UTC m=+1340.262706551"
Sep 10 16:39:53.027204 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:53.027131    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"
Sep 10 16:39:53.027204 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:53.027198    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"
Sep 10 16:39:58.948566 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:39:58.948525    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"]
Sep 10 16:40:00.157728 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:00.157695    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"]
Sep 10 16:40:07.565636 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.565604    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"]
Sep 10 16:40:07.568989 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.568971    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.571691 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.571661    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1925d8\"/\"rate-test-model-1925d8-kserve-self-signed-certs\""
Sep 10 16:40:07.577427 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.577396    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"]
Sep 10 16:40:07.671888 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.671862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52482454-547a-48f9-9cc9-6d73599a0dac-tls-certs\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.672004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.671895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-home\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.672004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.671916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-model-cache\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.672004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.671934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxkz2\" (UniqueName: \"kubernetes.io/projected/52482454-547a-48f9-9cc9-6d73599a0dac-kube-api-access-bxkz2\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.672004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.671959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-tmp-dir\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.672004 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.671985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-dshm\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.773226 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.773198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52482454-547a-48f9-9cc9-6d73599a0dac-tls-certs\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.773334 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.773236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-home\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.773334 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.773322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-model-cache\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.773422 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.773341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxkz2\" (UniqueName: \"kubernetes.io/projected/52482454-547a-48f9-9cc9-6d73599a0dac-kube-api-access-bxkz2\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.773422 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.773373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-tmp-dir\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.773422 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.773401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-dshm\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.773631 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.773607    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-home\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.773756 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.773653    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-model-cache\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.773756 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.773720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-tmp-dir\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.775723 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.775696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-dshm\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.775812 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.775797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52482454-547a-48f9-9cc9-6d73599a0dac-tls-certs\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.783150 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.783132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxkz2\" (UniqueName: \"kubernetes.io/projected/52482454-547a-48f9-9cc9-6d73599a0dac-kube-api-access-bxkz2\") pod \"rate-test-model-1925d8-kserve-64b848ccbc-d76t7\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") " pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:07.880523 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:07.880456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:08.153810 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:08.153782    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"]
Sep 10 16:40:08.155574 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:40:08.155542    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52482454_547a_48f9_9cc9_6d73599a0dac.slice/crio-74b1f8956bdf771f76351efa847dffe1e19dc0ec39780fe0e0b72a643711d6f1 WatchSource:0}: Error finding container 74b1f8956bdf771f76351efa847dffe1e19dc0ec39780fe0e0b72a643711d6f1: Status 404 returned error can't find the container with id 74b1f8956bdf771f76351efa847dffe1e19dc0ec39780fe0e0b72a643711d6f1
Sep 10 16:40:09.151906 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:09.151861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7" event={"ID":"52482454-547a-48f9-9cc9-6d73599a0dac","Type":"ContainerStarted","Data":"b90c8b5f6ad1aff9ef4f51d5934c2e26b32f2ef43e834dba3fca7800cdb35569"}
Sep 10 16:40:09.151906 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:09.151902    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7" event={"ID":"52482454-547a-48f9-9cc9-6d73599a0dac","Type":"ContainerStarted","Data":"74b1f8956bdf771f76351efa847dffe1e19dc0ec39780fe0e0b72a643711d6f1"}
Sep 10 16:40:09.182278 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:09.182230    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7" podStartSLOduration=2.182214521 podStartE2EDuration="2.182214521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:40:09.180818311 +0000 UTC m=+1367.381860723" watchObservedRunningTime="2026-09-10 16:40:09.182214521 +0000 UTC m=+1367.383256932"
Sep 10 16:40:13.967497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:13.967438    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw" podUID="491dd771-69c6-4282-81b9-d3aa5794620f" containerName="main" containerID="cri-o://1af19fe9b8b58e9e92b470efa6e6941a3361934b3418c81dd6a828e3251695e1" gracePeriod=45
Sep 10 16:40:14.177226 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.177194    2580 generic.go:356] "Generic (PLEG): container finished" podID="491dd771-69c6-4282-81b9-d3aa5794620f" containerID="1af19fe9b8b58e9e92b470efa6e6941a3361934b3418c81dd6a828e3251695e1" exitCode=0
Sep 10 16:40:14.177379 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.177243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw" event={"ID":"491dd771-69c6-4282-81b9-d3aa5794620f","Type":"ContainerDied","Data":"1af19fe9b8b58e9e92b470efa6e6941a3361934b3418c81dd6a828e3251695e1"}
Sep 10 16:40:14.219694 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.219629    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:40:14.326331 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.326302    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6fpg\" (UniqueName: \"kubernetes.io/projected/491dd771-69c6-4282-81b9-d3aa5794620f-kube-api-access-j6fpg\") pod \"491dd771-69c6-4282-81b9-d3aa5794620f\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") "
Sep 10 16:40:14.326558 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.326366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-model-cache\") pod \"491dd771-69c6-4282-81b9-d3aa5794620f\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") "
Sep 10 16:40:14.326558 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.326410    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-tmp-dir\") pod \"491dd771-69c6-4282-81b9-d3aa5794620f\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") "
Sep 10 16:40:14.326558 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.326500    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-home\") pod \"491dd771-69c6-4282-81b9-d3aa5794620f\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") "
Sep 10 16:40:14.326759 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.326578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/491dd771-69c6-4282-81b9-d3aa5794620f-tls-certs\") pod \"491dd771-69c6-4282-81b9-d3aa5794620f\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") "
Sep 10 16:40:14.326759 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.326643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-dshm\") pod \"491dd771-69c6-4282-81b9-d3aa5794620f\" (UID: \"491dd771-69c6-4282-81b9-d3aa5794620f\") "
Sep 10 16:40:14.326759 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.326645    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-model-cache" (OuterVolumeSpecName: "model-cache") pod "491dd771-69c6-4282-81b9-d3aa5794620f" (UID: "491dd771-69c6-4282-81b9-d3aa5794620f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:40:14.326759 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.326705    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "491dd771-69c6-4282-81b9-d3aa5794620f" (UID: "491dd771-69c6-4282-81b9-d3aa5794620f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:40:14.326759 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.326716    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-home" (OuterVolumeSpecName: "home") pod "491dd771-69c6-4282-81b9-d3aa5794620f" (UID: "491dd771-69c6-4282-81b9-d3aa5794620f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:40:14.327005 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.326941    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:40:14.327005 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.326956    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:40:14.327005 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.326963    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:40:14.328738 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.328710    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/491dd771-69c6-4282-81b9-d3aa5794620f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "491dd771-69c6-4282-81b9-d3aa5794620f" (UID: "491dd771-69c6-4282-81b9-d3aa5794620f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:40:14.328848 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.328768    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-dshm" (OuterVolumeSpecName: "dshm") pod "491dd771-69c6-4282-81b9-d3aa5794620f" (UID: "491dd771-69c6-4282-81b9-d3aa5794620f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:40:14.328953 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.328934    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491dd771-69c6-4282-81b9-d3aa5794620f-kube-api-access-j6fpg" (OuterVolumeSpecName: "kube-api-access-j6fpg") pod "491dd771-69c6-4282-81b9-d3aa5794620f" (UID: "491dd771-69c6-4282-81b9-d3aa5794620f"). InnerVolumeSpecName "kube-api-access-j6fpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:40:14.427406 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.427370    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/491dd771-69c6-4282-81b9-d3aa5794620f-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:40:14.427406 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.427401    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/491dd771-69c6-4282-81b9-d3aa5794620f-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:40:14.427406 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:14.427410    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6fpg\" (UniqueName: \"kubernetes.io/projected/491dd771-69c6-4282-81b9-d3aa5794620f-kube-api-access-j6fpg\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:40:15.176625 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.176575    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk" podUID="196a9276-b620-4f30-b77c-5e26b6b81e5a" containerName="main" containerID="cri-o://bc2fe43f8df8498a2409f3806b0f3eb88ef810c2d49fa025bb14c1497900f8e6" gracePeriod=45
Sep 10 16:40:15.184770 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.184746    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"
Sep 10 16:40:15.184915 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.184749    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw" event={"ID":"491dd771-69c6-4282-81b9-d3aa5794620f","Type":"ContainerDied","Data":"2979b5afe7d3903fdb95b43f7abe2323bc8f8a213b99381da911abe49c49e2b9"}
Sep 10 16:40:15.184915 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.184894    2580 scope.go:117] "RemoveContainer" containerID="1af19fe9b8b58e9e92b470efa6e6941a3361934b3418c81dd6a828e3251695e1"
Sep 10 16:40:15.267972 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.267947    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"]
Sep 10 16:40:15.275413 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.275391    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/sub-test-model-1925d8-kserve-6cf4f647f7-wkszw"]
Sep 10 16:40:15.427038 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.426981    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:40:15.537406 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.537378    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47l9p\" (UniqueName: \"kubernetes.io/projected/196a9276-b620-4f30-b77c-5e26b6b81e5a-kube-api-access-47l9p\") pod \"196a9276-b620-4f30-b77c-5e26b6b81e5a\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") "
Sep 10 16:40:15.537561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.537415    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-dshm\") pod \"196a9276-b620-4f30-b77c-5e26b6b81e5a\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") "
Sep 10 16:40:15.537561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.537445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-tmp-dir\") pod \"196a9276-b620-4f30-b77c-5e26b6b81e5a\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") "
Sep 10 16:40:15.537561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.537491    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-home\") pod \"196a9276-b620-4f30-b77c-5e26b6b81e5a\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") "
Sep 10 16:40:15.537785 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.537581    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-model-cache\") pod \"196a9276-b620-4f30-b77c-5e26b6b81e5a\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") "
Sep 10 16:40:15.537785 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.537626    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/196a9276-b620-4f30-b77c-5e26b6b81e5a-tls-certs\") pod \"196a9276-b620-4f30-b77c-5e26b6b81e5a\" (UID: \"196a9276-b620-4f30-b77c-5e26b6b81e5a\") "
Sep 10 16:40:15.537883 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.537784    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "196a9276-b620-4f30-b77c-5e26b6b81e5a" (UID: "196a9276-b620-4f30-b77c-5e26b6b81e5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:40:15.537883 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.537823    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-home" (OuterVolumeSpecName: "home") pod "196a9276-b620-4f30-b77c-5e26b6b81e5a" (UID: "196a9276-b620-4f30-b77c-5e26b6b81e5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:40:15.537883 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.537836    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "196a9276-b620-4f30-b77c-5e26b6b81e5a" (UID: "196a9276-b620-4f30-b77c-5e26b6b81e5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:40:15.538007 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.537931    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:40:15.538007 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.537945    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:40:15.538007 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.537957    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:40:15.539595 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.539574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-dshm" (OuterVolumeSpecName: "dshm") pod "196a9276-b620-4f30-b77c-5e26b6b81e5a" (UID: "196a9276-b620-4f30-b77c-5e26b6b81e5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:40:15.539681 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.539664    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196a9276-b620-4f30-b77c-5e26b6b81e5a-kube-api-access-47l9p" (OuterVolumeSpecName: "kube-api-access-47l9p") pod "196a9276-b620-4f30-b77c-5e26b6b81e5a" (UID: "196a9276-b620-4f30-b77c-5e26b6b81e5a"). InnerVolumeSpecName "kube-api-access-47l9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:40:15.539736 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.539663    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196a9276-b620-4f30-b77c-5e26b6b81e5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "196a9276-b620-4f30-b77c-5e26b6b81e5a" (UID: "196a9276-b620-4f30-b77c-5e26b6b81e5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:40:15.638688 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.638659    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/196a9276-b620-4f30-b77c-5e26b6b81e5a-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:40:15.638688 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.638684    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47l9p\" (UniqueName: \"kubernetes.io/projected/196a9276-b620-4f30-b77c-5e26b6b81e5a-kube-api-access-47l9p\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:40:15.638837 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:15.638696    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/196a9276-b620-4f30-b77c-5e26b6b81e5a-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:40:16.191946 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:16.190980    2580 generic.go:356] "Generic (PLEG): container finished" podID="196a9276-b620-4f30-b77c-5e26b6b81e5a" containerID="bc2fe43f8df8498a2409f3806b0f3eb88ef810c2d49fa025bb14c1497900f8e6" exitCode=0
Sep 10 16:40:16.191946 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:16.191167    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"
Sep 10 16:40:16.191946 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:16.191656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk" event={"ID":"196a9276-b620-4f30-b77c-5e26b6b81e5a","Type":"ContainerDied","Data":"bc2fe43f8df8498a2409f3806b0f3eb88ef810c2d49fa025bb14c1497900f8e6"}
Sep 10 16:40:16.191946 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:16.191695    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk" event={"ID":"196a9276-b620-4f30-b77c-5e26b6b81e5a","Type":"ContainerDied","Data":"7ac14c6425c91c70e94523ed4fc0189f668748438fbc656b158e655416d49ea1"}
Sep 10 16:40:16.191946 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:16.191719    2580 scope.go:117] "RemoveContainer" containerID="bc2fe43f8df8498a2409f3806b0f3eb88ef810c2d49fa025bb14c1497900f8e6"
Sep 10 16:40:16.204667 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:16.204639    2580 scope.go:117] "RemoveContainer" containerID="bc2fe43f8df8498a2409f3806b0f3eb88ef810c2d49fa025bb14c1497900f8e6"
Sep 10 16:40:16.206343 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:40:16.206316    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2fe43f8df8498a2409f3806b0f3eb88ef810c2d49fa025bb14c1497900f8e6\": container with ID starting with bc2fe43f8df8498a2409f3806b0f3eb88ef810c2d49fa025bb14c1497900f8e6 not found: ID does not exist" containerID="bc2fe43f8df8498a2409f3806b0f3eb88ef810c2d49fa025bb14c1497900f8e6"
Sep 10 16:40:16.206419 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:16.206356    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2fe43f8df8498a2409f3806b0f3eb88ef810c2d49fa025bb14c1497900f8e6"} err="failed to get container status \"bc2fe43f8df8498a2409f3806b0f3eb88ef810c2d49fa025bb14c1497900f8e6\": rpc error: code = NotFound desc = could not find container \"bc2fe43f8df8498a2409f3806b0f3eb88ef810c2d49fa025bb14c1497900f8e6\": container with ID starting with bc2fe43f8df8498a2409f3806b0f3eb88ef810c2d49fa025bb14c1497900f8e6 not found: ID does not exist"
Sep 10 16:40:16.226060 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:16.226036    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"]
Sep 10 16:40:16.231974 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:16.231952    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/sub-test-model-3ccbd4-kserve-779c8f6b5c-lmjsk"]
Sep 10 16:40:16.254397 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:16.254366    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="196a9276-b620-4f30-b77c-5e26b6b81e5a" path="/var/lib/kubelet/pods/196a9276-b620-4f30-b77c-5e26b6b81e5a/volumes"
Sep 10 16:40:16.254835 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:16.254819    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="491dd771-69c6-4282-81b9-d3aa5794620f" path="/var/lib/kubelet/pods/491dd771-69c6-4282-81b9-d3aa5794620f/volumes"
Sep 10 16:40:17.881041 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:17.881001    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:17.881491 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:17.881071    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:17.893796 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:17.893772    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:18.217899 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.217815    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:40:18.281559 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.281513    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"]
Sep 10 16:40:18.281993 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.281973    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="196a9276-b620-4f30-b77c-5e26b6b81e5a" containerName="main"
Sep 10 16:40:18.282089 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.281995    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="196a9276-b620-4f30-b77c-5e26b6b81e5a" containerName="main"
Sep 10 16:40:18.282089 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.282032    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="491dd771-69c6-4282-81b9-d3aa5794620f" containerName="main"
Sep 10 16:40:18.282089 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.282041    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="491dd771-69c6-4282-81b9-d3aa5794620f" containerName="main"
Sep 10 16:40:18.282252 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.282125    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="491dd771-69c6-4282-81b9-d3aa5794620f" containerName="main"
Sep 10 16:40:18.282252 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.282140    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="196a9276-b620-4f30-b77c-5e26b6b81e5a" containerName="main"
Sep 10 16:40:18.286449 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.286427    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"
Sep 10 16:40:18.288835 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.288815    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cdfd80-dockercfg-8mrkx\""
Sep 10 16:40:18.288941 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.288864    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cdfd80\""
Sep 10 16:40:18.295106 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.295085    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"]
Sep 10 16:40:18.357237 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.357206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f44e9dd-0645-4298-904a-a85726321dd6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld\" (UID: \"8f44e9dd-0645-4298-904a-a85726321dd6\") " pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"
Sep 10 16:40:18.357395 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.357279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjvhf\" (UniqueName: \"kubernetes.io/projected/8f44e9dd-0645-4298-904a-a85726321dd6-kube-api-access-qjvhf\") pod \"payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld\" (UID: \"8f44e9dd-0645-4298-904a-a85726321dd6\") " pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"
Sep 10 16:40:18.457925 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.457892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f44e9dd-0645-4298-904a-a85726321dd6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld\" (UID: \"8f44e9dd-0645-4298-904a-a85726321dd6\") " pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"
Sep 10 16:40:18.458053 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.457936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjvhf\" (UniqueName: \"kubernetes.io/projected/8f44e9dd-0645-4298-904a-a85726321dd6-kube-api-access-qjvhf\") pod \"payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld\" (UID: \"8f44e9dd-0645-4298-904a-a85726321dd6\") " pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"
Sep 10 16:40:18.458616 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.458596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f44e9dd-0645-4298-904a-a85726321dd6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld\" (UID: \"8f44e9dd-0645-4298-904a-a85726321dd6\") " pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"
Sep 10 16:40:18.470403 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.470335    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjvhf\" (UniqueName: \"kubernetes.io/projected/8f44e9dd-0645-4298-904a-a85726321dd6-kube-api-access-qjvhf\") pod \"payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld\" (UID: \"8f44e9dd-0645-4298-904a-a85726321dd6\") " pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"
Sep 10 16:40:18.597478 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.597449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"
Sep 10 16:40:18.878121 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:18.878097    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"]
Sep 10 16:40:18.879026 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:40:18.879003    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f44e9dd_0645_4298_904a_a85726321dd6.slice/crio-a9a58d5bc9c41e2f8d17ae1c29e85845da492634598ec402a32f8e6201bcdfd2 WatchSource:0}: Error finding container a9a58d5bc9c41e2f8d17ae1c29e85845da492634598ec402a32f8e6201bcdfd2: Status 404 returned error can't find the container with id a9a58d5bc9c41e2f8d17ae1c29e85845da492634598ec402a32f8e6201bcdfd2
Sep 10 16:40:19.211649 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:19.211564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld" event={"ID":"8f44e9dd-0645-4298-904a-a85726321dd6","Type":"ContainerStarted","Data":"fb0d0c993ace61dc600adbf61b1ab314c57d90b90b362a41f0c141d57d8a187f"}
Sep 10 16:40:19.212039 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:19.211765    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"
Sep 10 16:40:19.212039 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:19.211810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld" event={"ID":"8f44e9dd-0645-4298-904a-a85726321dd6","Type":"ContainerStarted","Data":"a9a58d5bc9c41e2f8d17ae1c29e85845da492634598ec402a32f8e6201bcdfd2"}
Sep 10 16:40:19.235790 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:19.235749    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld" podStartSLOduration=1.235734265 podStartE2EDuration="1.235734265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:40:19.23313638 +0000 UTC m=+1377.434178791" watchObservedRunningTime="2026-09-10 16:40:19.235734265 +0000 UTC m=+1377.436776655"
Sep 10 16:40:22.399644 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:22.399617    2580 scope.go:117] "RemoveContainer" containerID="cff42a9f1a787c58c2551f8cf2271e68a00328c763545dace6087beead0178ad"
Sep 10 16:40:22.407728 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:22.407706    2580 scope.go:117] "RemoveContainer" containerID="cc137dd849f8a94f2c40ec44e1919d11b4594758bd448ab13af39428553d4c4e"
Sep 10 16:40:30.217963 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:30.217909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"
Sep 10 16:40:44.370945 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:44.370907    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f"]
Sep 10 16:40:44.375872 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:44.375850    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f"
Sep 10 16:40:44.380527 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:44.380499    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:40:44.380527 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:44.380500    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\""
Sep 10 16:40:44.383678 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:44.383658    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f"]
Sep 10 16:40:44.457222 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:44.457189    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd4830d-8559-4454-bed4-44da937ba57c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f\" (UID: \"6bd4830d-8559-4454-bed4-44da937ba57c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f"
Sep 10 16:40:44.558083 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:44.558057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd4830d-8559-4454-bed4-44da937ba57c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f\" (UID: \"6bd4830d-8559-4454-bed4-44da937ba57c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f"
Sep 10 16:40:44.558675 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:44.558656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd4830d-8559-4454-bed4-44da937ba57c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f\" (UID: \"6bd4830d-8559-4454-bed4-44da937ba57c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f"
Sep 10 16:40:44.687557 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:44.687474    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f"
Sep 10 16:40:44.956978 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:44.956953    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f"]
Sep 10 16:40:44.958898 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:40:44.958875    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd4830d_8559_4454_bed4_44da937ba57c.slice/crio-cc719187d47f6bc5b909ff8d0ce1abe77e2711c03eaaef68f58a848a08d40546 WatchSource:0}: Error finding container cc719187d47f6bc5b909ff8d0ce1abe77e2711c03eaaef68f58a848a08d40546: Status 404 returned error can't find the container with id cc719187d47f6bc5b909ff8d0ce1abe77e2711c03eaaef68f58a848a08d40546
Sep 10 16:40:45.328384 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:45.328352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f" event={"ID":"6bd4830d-8559-4454-bed4-44da937ba57c","Type":"ContainerStarted","Data":"3697714744df2340c275e8af989dfc0b0d62aae211e1c004c9f0e681d10ec409"}
Sep 10 16:40:45.328596 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:45.328402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f" event={"ID":"6bd4830d-8559-4454-bed4-44da937ba57c","Type":"ContainerStarted","Data":"cc719187d47f6bc5b909ff8d0ce1abe77e2711c03eaaef68f58a848a08d40546"}
Sep 10 16:40:45.352959 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:45.352908    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f" podStartSLOduration=1.352892238 podStartE2EDuration="1.352892238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:40:45.351458615 +0000 UTC m=+1403.552501027" watchObservedRunningTime="2026-09-10 16:40:45.352892238 +0000 UTC m=+1403.553934649"
Sep 10 16:40:46.333965 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:46.333933    2580 generic.go:356] "Generic (PLEG): container finished" podID="6bd4830d-8559-4454-bed4-44da937ba57c" containerID="3697714744df2340c275e8af989dfc0b0d62aae211e1c004c9f0e681d10ec409" exitCode=0
Sep 10 16:40:46.334341 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:46.333986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f" event={"ID":"6bd4830d-8559-4454-bed4-44da937ba57c","Type":"ContainerDied","Data":"3697714744df2340c275e8af989dfc0b0d62aae211e1c004c9f0e681d10ec409"}
Sep 10 16:40:47.469352 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:47.469327    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f"
Sep 10 16:40:47.584898 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:47.584868    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd4830d-8559-4454-bed4-44da937ba57c-maas-api-service-ca\") pod \"6bd4830d-8559-4454-bed4-44da937ba57c\" (UID: \"6bd4830d-8559-4454-bed4-44da937ba57c\") "
Sep 10 16:40:47.585241 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:47.585214    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd4830d-8559-4454-bed4-44da937ba57c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6bd4830d-8559-4454-bed4-44da937ba57c" (UID: "6bd4830d-8559-4454-bed4-44da937ba57c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:40:47.685732 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:47.685663    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd4830d-8559-4454-bed4-44da937ba57c-maas-api-service-ca\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:40:48.343832 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:48.343807    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f"
Sep 10 16:40:48.343984 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:48.343800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f" event={"ID":"6bd4830d-8559-4454-bed4-44da937ba57c","Type":"ContainerDied","Data":"cc719187d47f6bc5b909ff8d0ce1abe77e2711c03eaaef68f58a848a08d40546"}
Sep 10 16:40:48.343984 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:48.343923    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc719187d47f6bc5b909ff8d0ce1abe77e2711c03eaaef68f58a848a08d40546"
Sep 10 16:40:52.558095 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.558067    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"]
Sep 10 16:40:52.558447 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.558422    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd4830d-8559-4454-bed4-44da937ba57c" containerName="revoke-keys"
Sep 10 16:40:52.558447 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.558435    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd4830d-8559-4454-bed4-44da937ba57c" containerName="revoke-keys"
Sep 10 16:40:52.558520 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.558503    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd4830d-8559-4454-bed4-44da937ba57c" containerName="revoke-keys"
Sep 10 16:40:52.561476 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.561460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.564624 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.564599    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ccbd4\"/\"kube-root-ca.crt\""
Sep 10 16:40:52.564735 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.564648    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ccbd4\"/\"default-dockercfg-lpb8g\""
Sep 10 16:40:52.565214 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.565196    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ccbd4\"/\"openshift-service-ca.crt\""
Sep 10 16:40:52.565526 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.565509    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ccbd4\"/\"rate-test-model-3ccbd4-kserve-self-signed-certs\""
Sep 10 16:40:52.575612 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.575587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"]
Sep 10 16:40:52.628095 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.628068    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-model-cache\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.628200 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.628103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82ee96c5-2189-44f5-bbe9-eca0b013a28f-tls-certs\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.628200 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.628133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjx5k\" (UniqueName: \"kubernetes.io/projected/82ee96c5-2189-44f5-bbe9-eca0b013a28f-kube-api-access-wjx5k\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.628283 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.628206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-home\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.628283 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.628248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-tmp-dir\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.628346 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.628279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-dshm\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.729310 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.729260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjx5k\" (UniqueName: \"kubernetes.io/projected/82ee96c5-2189-44f5-bbe9-eca0b013a28f-kube-api-access-wjx5k\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.729310 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.729317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-home\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.729555 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.729353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-tmp-dir\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.729555 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.729382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-dshm\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.729555 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.729439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-model-cache\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.729555 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.729477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82ee96c5-2189-44f5-bbe9-eca0b013a28f-tls-certs\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.729863 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.729838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-tmp-dir\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.729925 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.729908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-model-cache\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.729984 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.729913    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-home\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.731846 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.731822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-dshm\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.732066 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.732047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82ee96c5-2189-44f5-bbe9-eca0b013a28f-tls-certs\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.739553 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.739519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjx5k\" (UniqueName: \"kubernetes.io/projected/82ee96c5-2189-44f5-bbe9-eca0b013a28f-kube-api-access-wjx5k\") pod \"rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") " pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:52.870742 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:52.870681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:40:53.143559 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:53.143520    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"]
Sep 10 16:40:53.146183 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:40:53.146158    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82ee96c5_2189_44f5_bbe9_eca0b013a28f.slice/crio-dec2b46ab1b5cdcbfcffee3b2b5dce5f35c63ffb4546039d2f856a0f824df1ec WatchSource:0}: Error finding container dec2b46ab1b5cdcbfcffee3b2b5dce5f35c63ffb4546039d2f856a0f824df1ec: Status 404 returned error can't find the container with id dec2b46ab1b5cdcbfcffee3b2b5dce5f35c63ffb4546039d2f856a0f824df1ec
Sep 10 16:40:53.368716 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:53.368679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd" event={"ID":"82ee96c5-2189-44f5-bbe9-eca0b013a28f","Type":"ContainerStarted","Data":"7d2b93a551b8b1ccabeeba989111af3f2d2d12a990cff3b2fa3205556e0a90e4"}
Sep 10 16:40:53.368716 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:53.368719    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd" event={"ID":"82ee96c5-2189-44f5-bbe9-eca0b013a28f","Type":"ContainerStarted","Data":"dec2b46ab1b5cdcbfcffee3b2b5dce5f35c63ffb4546039d2f856a0f824df1ec"}
Sep 10 16:40:53.389055 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:40:53.389002    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd" podStartSLOduration=1.388987081 podStartE2EDuration="1.388987081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:40:53.387474407 +0000 UTC m=+1411.588516817" watchObservedRunningTime="2026-09-10 16:40:53.388987081 +0000 UTC m=+1411.590029491"
Sep 10 16:41:00.361551 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.361499    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"]
Sep 10 16:41:00.362003 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.361767    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng" podUID="cb20ecaa-57f4-4393-8091-bf8fcfc14937" containerName="payload-processing" containerID="cri-o://ceb0001b7be091b309e685be56e57f0af50cdb5db82a569d78439a4c1d5ef938" gracePeriod=30
Sep 10 16:41:00.373226 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.373197    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"]
Sep 10 16:41:00.373480 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.373432    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq" podUID="2477c2fa-f4e8-458c-b5be-09af9f69aa88" containerName="payload-pre-processing" containerID="cri-o://043c880dafc8dc67c63d3abd32d8806c7fac037e998d734ec07d87d17e977676" gracePeriod=30
Sep 10 16:41:00.619225 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.619163    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"
Sep 10 16:41:00.648825 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.648805    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"
Sep 10 16:41:00.686036 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.686014    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f"]
Sep 10 16:41:00.696568 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.696545    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqwkv\" (UniqueName: \"kubernetes.io/projected/cb20ecaa-57f4-4393-8091-bf8fcfc14937-kube-api-access-xqwkv\") pod \"cb20ecaa-57f4-4393-8091-bf8fcfc14937\" (UID: \"cb20ecaa-57f4-4393-8091-bf8fcfc14937\") "
Sep 10 16:41:00.696674 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.696623    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb20ecaa-57f4-4393-8091-bf8fcfc14937-plugins-config-volume\") pod \"cb20ecaa-57f4-4393-8091-bf8fcfc14937\" (UID: \"cb20ecaa-57f4-4393-8091-bf8fcfc14937\") "
Sep 10 16:41:00.697176 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.697117    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb20ecaa-57f4-4393-8091-bf8fcfc14937-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb20ecaa-57f4-4393-8091-bf8fcfc14937" (UID: "cb20ecaa-57f4-4393-8091-bf8fcfc14937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:41:00.698691 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.698662    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb20ecaa-57f4-4393-8091-bf8fcfc14937-kube-api-access-xqwkv" (OuterVolumeSpecName: "kube-api-access-xqwkv") pod "cb20ecaa-57f4-4393-8091-bf8fcfc14937" (UID: "cb20ecaa-57f4-4393-8091-bf8fcfc14937"). InnerVolumeSpecName "kube-api-access-xqwkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:41:00.700276 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.700252    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fb3d69-whv4f"]
Sep 10 16:41:00.797384 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.797360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2477c2fa-f4e8-458c-b5be-09af9f69aa88-plugins-config-volume\") pod \"2477c2fa-f4e8-458c-b5be-09af9f69aa88\" (UID: \"2477c2fa-f4e8-458c-b5be-09af9f69aa88\") "
Sep 10 16:41:00.797555 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.797393    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z79nh\" (UniqueName: \"kubernetes.io/projected/2477c2fa-f4e8-458c-b5be-09af9f69aa88-kube-api-access-z79nh\") pod \"2477c2fa-f4e8-458c-b5be-09af9f69aa88\" (UID: \"2477c2fa-f4e8-458c-b5be-09af9f69aa88\") "
Sep 10 16:41:00.797632 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.797603    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb20ecaa-57f4-4393-8091-bf8fcfc14937-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:00.797632 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.797614    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqwkv\" (UniqueName: \"kubernetes.io/projected/cb20ecaa-57f4-4393-8091-bf8fcfc14937-kube-api-access-xqwkv\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:00.797819 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.797797    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2477c2fa-f4e8-458c-b5be-09af9f69aa88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2477c2fa-f4e8-458c-b5be-09af9f69aa88" (UID: "2477c2fa-f4e8-458c-b5be-09af9f69aa88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:41:00.799420 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.799402    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2477c2fa-f4e8-458c-b5be-09af9f69aa88-kube-api-access-z79nh" (OuterVolumeSpecName: "kube-api-access-z79nh") pod "2477c2fa-f4e8-458c-b5be-09af9f69aa88" (UID: "2477c2fa-f4e8-458c-b5be-09af9f69aa88"). InnerVolumeSpecName "kube-api-access-z79nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:41:00.898105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.898049    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2477c2fa-f4e8-458c-b5be-09af9f69aa88-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:00.898105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:00.898075    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z79nh\" (UniqueName: \"kubernetes.io/projected/2477c2fa-f4e8-458c-b5be-09af9f69aa88-kube-api-access-z79nh\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:01.406596 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.406558    2580 generic.go:356] "Generic (PLEG): container finished" podID="cb20ecaa-57f4-4393-8091-bf8fcfc14937" containerID="ceb0001b7be091b309e685be56e57f0af50cdb5db82a569d78439a4c1d5ef938" exitCode=0
Sep 10 16:41:01.406997 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.406612    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"
Sep 10 16:41:01.406997 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.406630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng" event={"ID":"cb20ecaa-57f4-4393-8091-bf8fcfc14937","Type":"ContainerDied","Data":"ceb0001b7be091b309e685be56e57f0af50cdb5db82a569d78439a4c1d5ef938"}
Sep 10 16:41:01.406997 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.406687    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng" event={"ID":"cb20ecaa-57f4-4393-8091-bf8fcfc14937","Type":"ContainerDied","Data":"441adf06ea5a9b024fbc93faa98785fe19d6e53d9d3324e23ac9901bfeeb3092"}
Sep 10 16:41:01.406997 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.406717    2580 scope.go:117] "RemoveContainer" containerID="ceb0001b7be091b309e685be56e57f0af50cdb5db82a569d78439a4c1d5ef938"
Sep 10 16:41:01.408030 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.408009    2580 generic.go:356] "Generic (PLEG): container finished" podID="2477c2fa-f4e8-458c-b5be-09af9f69aa88" containerID="043c880dafc8dc67c63d3abd32d8806c7fac037e998d734ec07d87d17e977676" exitCode=0
Sep 10 16:41:01.408111 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.408051    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"
Sep 10 16:41:01.408111 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.408055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq" event={"ID":"2477c2fa-f4e8-458c-b5be-09af9f69aa88","Type":"ContainerDied","Data":"043c880dafc8dc67c63d3abd32d8806c7fac037e998d734ec07d87d17e977676"}
Sep 10 16:41:01.408206 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.408123    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq" event={"ID":"2477c2fa-f4e8-458c-b5be-09af9f69aa88","Type":"ContainerDied","Data":"00509f859655bc2e0318b3388f4ec8ae8f6356353eec466addaf28302474e5e5"}
Sep 10 16:41:01.415705 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.415685    2580 scope.go:117] "RemoveContainer" containerID="ceb0001b7be091b309e685be56e57f0af50cdb5db82a569d78439a4c1d5ef938"
Sep 10 16:41:01.415969 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:41:01.415948    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb0001b7be091b309e685be56e57f0af50cdb5db82a569d78439a4c1d5ef938\": container with ID starting with ceb0001b7be091b309e685be56e57f0af50cdb5db82a569d78439a4c1d5ef938 not found: ID does not exist" containerID="ceb0001b7be091b309e685be56e57f0af50cdb5db82a569d78439a4c1d5ef938"
Sep 10 16:41:01.416054 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.415978    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb0001b7be091b309e685be56e57f0af50cdb5db82a569d78439a4c1d5ef938"} err="failed to get container status \"ceb0001b7be091b309e685be56e57f0af50cdb5db82a569d78439a4c1d5ef938\": rpc error: code = NotFound desc = could not find container \"ceb0001b7be091b309e685be56e57f0af50cdb5db82a569d78439a4c1d5ef938\": container with ID starting with ceb0001b7be091b309e685be56e57f0af50cdb5db82a569d78439a4c1d5ef938 not found: ID does not exist"
Sep 10 16:41:01.416054 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.415995    2580 scope.go:117] "RemoveContainer" containerID="043c880dafc8dc67c63d3abd32d8806c7fac037e998d734ec07d87d17e977676"
Sep 10 16:41:01.423813 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.423798    2580 scope.go:117] "RemoveContainer" containerID="043c880dafc8dc67c63d3abd32d8806c7fac037e998d734ec07d87d17e977676"
Sep 10 16:41:01.424062 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:41:01.424042    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043c880dafc8dc67c63d3abd32d8806c7fac037e998d734ec07d87d17e977676\": container with ID starting with 043c880dafc8dc67c63d3abd32d8806c7fac037e998d734ec07d87d17e977676 not found: ID does not exist" containerID="043c880dafc8dc67c63d3abd32d8806c7fac037e998d734ec07d87d17e977676"
Sep 10 16:41:01.424117 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.424073    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043c880dafc8dc67c63d3abd32d8806c7fac037e998d734ec07d87d17e977676"} err="failed to get container status \"043c880dafc8dc67c63d3abd32d8806c7fac037e998d734ec07d87d17e977676\": rpc error: code = NotFound desc = could not find container \"043c880dafc8dc67c63d3abd32d8806c7fac037e998d734ec07d87d17e977676\": container with ID starting with 043c880dafc8dc67c63d3abd32d8806c7fac037e998d734ec07d87d17e977676 not found: ID does not exist"
Sep 10 16:41:01.435221 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.435202    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"]
Sep 10 16:41:01.440750 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.440731    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fb3d69-8c5c85b98-8hwfq"]
Sep 10 16:41:01.459278 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.459256    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"]
Sep 10 16:41:01.478705 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:01.478682    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fb3d69-5549989f8c-4gwng"]
Sep 10 16:41:02.254780 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:02.254748    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2477c2fa-f4e8-458c-b5be-09af9f69aa88" path="/var/lib/kubelet/pods/2477c2fa-f4e8-458c-b5be-09af9f69aa88/volumes"
Sep 10 16:41:02.255052 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:02.255040    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bd4830d-8559-4454-bed4-44da937ba57c" path="/var/lib/kubelet/pods/6bd4830d-8559-4454-bed4-44da937ba57c/volumes"
Sep 10 16:41:02.255348 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:02.255337    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb20ecaa-57f4-4393-8091-bf8fcfc14937" path="/var/lib/kubelet/pods/cb20ecaa-57f4-4393-8091-bf8fcfc14937/volumes"
Sep 10 16:41:02.871020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:02.870989    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:41:02.871411 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:02.871229    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:41:02.883973 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:02.883947    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:41:03.430286 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:03.430257    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:41:10.894564 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:10.894524    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8"]
Sep 10 16:41:10.895041 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:10.895022    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2477c2fa-f4e8-458c-b5be-09af9f69aa88" containerName="payload-pre-processing"
Sep 10 16:41:10.895121 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:10.895045    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2477c2fa-f4e8-458c-b5be-09af9f69aa88" containerName="payload-pre-processing"
Sep 10 16:41:10.895121 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:10.895058    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb20ecaa-57f4-4393-8091-bf8fcfc14937" containerName="payload-processing"
Sep 10 16:41:10.895121 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:10.895066    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb20ecaa-57f4-4393-8091-bf8fcfc14937" containerName="payload-processing"
Sep 10 16:41:10.895262 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:10.895165    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb20ecaa-57f4-4393-8091-bf8fcfc14937" containerName="payload-processing"
Sep 10 16:41:10.895262 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:10.895178    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2477c2fa-f4e8-458c-b5be-09af9f69aa88" containerName="payload-pre-processing"
Sep 10 16:41:10.899726 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:10.899704    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8"
Sep 10 16:41:10.906655 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:10.906635    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\""
Sep 10 16:41:10.906760 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:10.906635    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:41:10.911596 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:10.911575    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8"]
Sep 10 16:41:11.071179 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:11.071153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98ae61-7712-49c8-babe-1b471479981b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8\" (UID: \"6a98ae61-7712-49c8-babe-1b471479981b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8"
Sep 10 16:41:11.171954 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:11.171892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98ae61-7712-49c8-babe-1b471479981b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8\" (UID: \"6a98ae61-7712-49c8-babe-1b471479981b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8"
Sep 10 16:41:11.172451 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:11.172433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98ae61-7712-49c8-babe-1b471479981b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8\" (UID: \"6a98ae61-7712-49c8-babe-1b471479981b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8"
Sep 10 16:41:11.209308 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:11.209288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8"
Sep 10 16:41:11.493377 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:41:11.493341    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a98ae61_7712_49c8_babe_1b471479981b.slice/crio-16060a6862a1a7bbde609dd62642be12b7fa40f3d2f9965b9070a2e76663af4d WatchSource:0}: Error finding container 16060a6862a1a7bbde609dd62642be12b7fa40f3d2f9965b9070a2e76663af4d: Status 404 returned error can't find the container with id 16060a6862a1a7bbde609dd62642be12b7fa40f3d2f9965b9070a2e76663af4d
Sep 10 16:41:11.495148 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:11.495120    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8"]
Sep 10 16:41:12.461373 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:12.461298    2580 generic.go:356] "Generic (PLEG): container finished" podID="6a98ae61-7712-49c8-babe-1b471479981b" containerID="eeffdeb6ec9dca8370f269f704560312dc3e4e51126f16f36f6e43045229b317" exitCode=0
Sep 10 16:41:12.461373 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:12.461362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8" event={"ID":"6a98ae61-7712-49c8-babe-1b471479981b","Type":"ContainerDied","Data":"eeffdeb6ec9dca8370f269f704560312dc3e4e51126f16f36f6e43045229b317"}
Sep 10 16:41:12.461764 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:12.461410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8" event={"ID":"6a98ae61-7712-49c8-babe-1b471479981b","Type":"ContainerStarted","Data":"16060a6862a1a7bbde609dd62642be12b7fa40f3d2f9965b9070a2e76663af4d"}
Sep 10 16:41:13.595733 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:13.595707    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8"
Sep 10 16:41:13.693489 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:13.693461    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98ae61-7712-49c8-babe-1b471479981b-maas-api-service-ca\") pod \"6a98ae61-7712-49c8-babe-1b471479981b\" (UID: \"6a98ae61-7712-49c8-babe-1b471479981b\") "
Sep 10 16:41:13.693794 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:13.693772    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a98ae61-7712-49c8-babe-1b471479981b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a98ae61-7712-49c8-babe-1b471479981b" (UID: "6a98ae61-7712-49c8-babe-1b471479981b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:41:13.794428 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:13.794403    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98ae61-7712-49c8-babe-1b471479981b-maas-api-service-ca\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:14.472180 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:14.472150    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8"
Sep 10 16:41:14.472368 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:14.472190    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8" event={"ID":"6a98ae61-7712-49c8-babe-1b471479981b","Type":"ContainerDied","Data":"16060a6862a1a7bbde609dd62642be12b7fa40f3d2f9965b9070a2e76663af4d"}
Sep 10 16:41:14.472368 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:14.472245    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16060a6862a1a7bbde609dd62642be12b7fa40f3d2f9965b9070a2e76663af4d"
Sep 10 16:41:26.685194 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:26.685161    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"]
Sep 10 16:41:26.685704 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:26.685357    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld" podUID="8f44e9dd-0645-4298-904a-a85726321dd6" containerName="payload-processing" containerID="cri-o://fb0d0c993ace61dc600adbf61b1ab314c57d90b90b362a41f0c141d57d8a187f" gracePeriod=30
Sep 10 16:41:26.940204 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:26.940147    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"
Sep 10 16:41:26.976888 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:26.976852    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8"]
Sep 10 16:41:26.980062 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:26.980039    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cdfd80-znrm8"]
Sep 10 16:41:27.003341 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.003318    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjvhf\" (UniqueName: \"kubernetes.io/projected/8f44e9dd-0645-4298-904a-a85726321dd6-kube-api-access-qjvhf\") pod \"8f44e9dd-0645-4298-904a-a85726321dd6\" (UID: \"8f44e9dd-0645-4298-904a-a85726321dd6\") "
Sep 10 16:41:27.003458 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.003431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f44e9dd-0645-4298-904a-a85726321dd6-plugins-config-volume\") pod \"8f44e9dd-0645-4298-904a-a85726321dd6\" (UID: \"8f44e9dd-0645-4298-904a-a85726321dd6\") "
Sep 10 16:41:27.003829 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.003808    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f44e9dd-0645-4298-904a-a85726321dd6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f44e9dd-0645-4298-904a-a85726321dd6" (UID: "8f44e9dd-0645-4298-904a-a85726321dd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:41:27.005386 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.005364    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f44e9dd-0645-4298-904a-a85726321dd6-kube-api-access-qjvhf" (OuterVolumeSpecName: "kube-api-access-qjvhf") pod "8f44e9dd-0645-4298-904a-a85726321dd6" (UID: "8f44e9dd-0645-4298-904a-a85726321dd6"). InnerVolumeSpecName "kube-api-access-qjvhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:41:27.104112 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.104076    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f44e9dd-0645-4298-904a-a85726321dd6-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:27.104112 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.104105    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjvhf\" (UniqueName: \"kubernetes.io/projected/8f44e9dd-0645-4298-904a-a85726321dd6-kube-api-access-qjvhf\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:27.535268 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.535235    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f44e9dd-0645-4298-904a-a85726321dd6" containerID="fb0d0c993ace61dc600adbf61b1ab314c57d90b90b362a41f0c141d57d8a187f" exitCode=0
Sep 10 16:41:27.535459 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.535283    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"
Sep 10 16:41:27.535459 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.535317    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld" event={"ID":"8f44e9dd-0645-4298-904a-a85726321dd6","Type":"ContainerDied","Data":"fb0d0c993ace61dc600adbf61b1ab314c57d90b90b362a41f0c141d57d8a187f"}
Sep 10 16:41:27.535459 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.535373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld" event={"ID":"8f44e9dd-0645-4298-904a-a85726321dd6","Type":"ContainerDied","Data":"a9a58d5bc9c41e2f8d17ae1c29e85845da492634598ec402a32f8e6201bcdfd2"}
Sep 10 16:41:27.535459 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.535396    2580 scope.go:117] "RemoveContainer" containerID="fb0d0c993ace61dc600adbf61b1ab314c57d90b90b362a41f0c141d57d8a187f"
Sep 10 16:41:27.543990 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.543964    2580 scope.go:117] "RemoveContainer" containerID="fb0d0c993ace61dc600adbf61b1ab314c57d90b90b362a41f0c141d57d8a187f"
Sep 10 16:41:27.544281 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:41:27.544264    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0d0c993ace61dc600adbf61b1ab314c57d90b90b362a41f0c141d57d8a187f\": container with ID starting with fb0d0c993ace61dc600adbf61b1ab314c57d90b90b362a41f0c141d57d8a187f not found: ID does not exist" containerID="fb0d0c993ace61dc600adbf61b1ab314c57d90b90b362a41f0c141d57d8a187f"
Sep 10 16:41:27.544351 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.544288    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0d0c993ace61dc600adbf61b1ab314c57d90b90b362a41f0c141d57d8a187f"} err="failed to get container status \"fb0d0c993ace61dc600adbf61b1ab314c57d90b90b362a41f0c141d57d8a187f\": rpc error: code = NotFound desc = could not find container \"fb0d0c993ace61dc600adbf61b1ab314c57d90b90b362a41f0c141d57d8a187f\": container with ID starting with fb0d0c993ace61dc600adbf61b1ab314c57d90b90b362a41f0c141d57d8a187f not found: ID does not exist"
Sep 10 16:41:27.563509 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.563487    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"]
Sep 10 16:41:27.568850 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:27.568831    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cdfd80-5c467cb7c5-2kxld"]
Sep 10 16:41:28.254494 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:28.254462    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a98ae61-7712-49c8-babe-1b471479981b" path="/var/lib/kubelet/pods/6a98ae61-7712-49c8-babe-1b471479981b/volumes"
Sep 10 16:41:28.254863 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:28.254849    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f44e9dd-0645-4298-904a-a85726321dd6" path="/var/lib/kubelet/pods/8f44e9dd-0645-4298-904a-a85726321dd6/volumes"
Sep 10 16:41:35.870295 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:35.870262    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4"]
Sep 10 16:41:35.870692 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:35.870577    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a98ae61-7712-49c8-babe-1b471479981b" containerName="revoke-keys"
Sep 10 16:41:35.870692 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:35.870591    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a98ae61-7712-49c8-babe-1b471479981b" containerName="revoke-keys"
Sep 10 16:41:35.870692 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:35.870610    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f44e9dd-0645-4298-904a-a85726321dd6" containerName="payload-processing"
Sep 10 16:41:35.870692 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:35.870616    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f44e9dd-0645-4298-904a-a85726321dd6" containerName="payload-processing"
Sep 10 16:41:35.870692 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:35.870671    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a98ae61-7712-49c8-babe-1b471479981b" containerName="revoke-keys"
Sep 10 16:41:35.870692 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:35.870682    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f44e9dd-0645-4298-904a-a85726321dd6" containerName="payload-processing"
Sep 10 16:41:35.874817 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:35.874800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4"
Sep 10 16:41:35.877398 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:35.877334    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\""
Sep 10 16:41:35.877540 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:35.877510    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:41:35.883078 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:35.883057    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4"]
Sep 10 16:41:35.975213 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:35.975188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c310823-0d87-4860-8d1f-b84820b7b3a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4\" (UID: \"6c310823-0d87-4860-8d1f-b84820b7b3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4"
Sep 10 16:41:36.076201 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:36.076171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c310823-0d87-4860-8d1f-b84820b7b3a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4\" (UID: \"6c310823-0d87-4860-8d1f-b84820b7b3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4"
Sep 10 16:41:36.076808 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:36.076789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c310823-0d87-4860-8d1f-b84820b7b3a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4\" (UID: \"6c310823-0d87-4860-8d1f-b84820b7b3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4"
Sep 10 16:41:36.185520 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:36.185432    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4"
Sep 10 16:41:36.455369 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:36.455344    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4"]
Sep 10 16:41:36.457001 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:41:36.456975    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c310823_0d87_4860_8d1f_b84820b7b3a6.slice/crio-bd956cf72aef469e4f5815a1506d93fdcd9fa3f90d599da13a7d3d3eb9885fcf WatchSource:0}: Error finding container bd956cf72aef469e4f5815a1506d93fdcd9fa3f90d599da13a7d3d3eb9885fcf: Status 404 returned error can't find the container with id bd956cf72aef469e4f5815a1506d93fdcd9fa3f90d599da13a7d3d3eb9885fcf
Sep 10 16:41:36.579496 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:36.579465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4" event={"ID":"6c310823-0d87-4860-8d1f-b84820b7b3a6","Type":"ContainerStarted","Data":"bd956cf72aef469e4f5815a1506d93fdcd9fa3f90d599da13a7d3d3eb9885fcf"}
Sep 10 16:41:37.286977 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:37.286884    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"]
Sep 10 16:41:37.585127 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:37.585047    2580 generic.go:356] "Generic (PLEG): container finished" podID="6c310823-0d87-4860-8d1f-b84820b7b3a6" containerID="c2c1c34a4de3cd70c4f6a13b6a4c777bd5e11bbc421aaa0c13bd90d644989999" exitCode=0
Sep 10 16:41:37.585259 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:37.585132    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4" event={"ID":"6c310823-0d87-4860-8d1f-b84820b7b3a6","Type":"ContainerDied","Data":"c2c1c34a4de3cd70c4f6a13b6a4c777bd5e11bbc421aaa0c13bd90d644989999"}
Sep 10 16:41:38.236394 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:38.236360    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"]
Sep 10 16:41:38.719393 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:38.719371    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4"
Sep 10 16:41:38.794973 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:38.794946    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c310823-0d87-4860-8d1f-b84820b7b3a6-maas-api-service-ca\") pod \"6c310823-0d87-4860-8d1f-b84820b7b3a6\" (UID: \"6c310823-0d87-4860-8d1f-b84820b7b3a6\") "
Sep 10 16:41:38.795263 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:38.795240    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c310823-0d87-4860-8d1f-b84820b7b3a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c310823-0d87-4860-8d1f-b84820b7b3a6" (UID: "6c310823-0d87-4860-8d1f-b84820b7b3a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:41:38.896479 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:38.896418    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c310823-0d87-4860-8d1f-b84820b7b3a6-maas-api-service-ca\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:39.596787 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:39.596756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4" event={"ID":"6c310823-0d87-4860-8d1f-b84820b7b3a6","Type":"ContainerDied","Data":"bd956cf72aef469e4f5815a1506d93fdcd9fa3f90d599da13a7d3d3eb9885fcf"}
Sep 10 16:41:39.596787 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:39.596774    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4"
Sep 10 16:41:39.596993 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:39.596804    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd956cf72aef469e4f5815a1506d93fdcd9fa3f90d599da13a7d3d3eb9885fcf"
Sep 10 16:41:46.635294 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.635256    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"]
Sep 10 16:41:46.635817 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.635510    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2" podUID="f87220fa-3685-4b19-ba33-538f662d6871" containerName="payload-processing" containerID="cri-o://494eaa647581b088d44669e4ec96112ae21beace8d9c681fe5ab80fd5d7a824c" gracePeriod=30
Sep 10 16:41:46.651177 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.651150    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"]
Sep 10 16:41:46.651434 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.651403    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7" podUID="f71e1269-02e8-45e7-83be-0c1706594a47" containerName="payload-pre-processing" containerID="cri-o://85be887bf503b577f80a27ba86f37230f9ae9d67b51942a84a4bf07cf3987cf0" gracePeriod=30
Sep 10 16:41:46.675095 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.675064    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2" podUID="f87220fa-3685-4b19-ba33-538f662d6871" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.45:9004: connect: connection refused"
Sep 10 16:41:46.675095 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.675076    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7" podUID="f71e1269-02e8-45e7-83be-0c1706594a47" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.46:9004: connect: connection refused"
Sep 10 16:41:46.838071 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.838043    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4"]
Sep 10 16:41:46.843337 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.843312    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd1ece-br7t4"]
Sep 10 16:41:46.915109 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.915088    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"
Sep 10 16:41:46.917759 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.917738    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"
Sep 10 16:41:46.954646 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.954620    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc7sg\" (UniqueName: \"kubernetes.io/projected/f87220fa-3685-4b19-ba33-538f662d6871-kube-api-access-vc7sg\") pod \"f87220fa-3685-4b19-ba33-538f662d6871\" (UID: \"f87220fa-3685-4b19-ba33-538f662d6871\") "
Sep 10 16:41:46.954743 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.954675    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhdgg\" (UniqueName: \"kubernetes.io/projected/f71e1269-02e8-45e7-83be-0c1706594a47-kube-api-access-nhdgg\") pod \"f71e1269-02e8-45e7-83be-0c1706594a47\" (UID: \"f71e1269-02e8-45e7-83be-0c1706594a47\") "
Sep 10 16:41:46.954786 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.954745    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87220fa-3685-4b19-ba33-538f662d6871-plugins-config-volume\") pod \"f87220fa-3685-4b19-ba33-538f662d6871\" (UID: \"f87220fa-3685-4b19-ba33-538f662d6871\") "
Sep 10 16:41:46.954822 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.954789    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71e1269-02e8-45e7-83be-0c1706594a47-plugins-config-volume\") pod \"f71e1269-02e8-45e7-83be-0c1706594a47\" (UID: \"f71e1269-02e8-45e7-83be-0c1706594a47\") "
Sep 10 16:41:46.955192 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.955165    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87220fa-3685-4b19-ba33-538f662d6871-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f87220fa-3685-4b19-ba33-538f662d6871" (UID: "f87220fa-3685-4b19-ba33-538f662d6871"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:41:46.955283 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.955256    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71e1269-02e8-45e7-83be-0c1706594a47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f71e1269-02e8-45e7-83be-0c1706594a47" (UID: "f71e1269-02e8-45e7-83be-0c1706594a47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:41:46.956787 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.956767    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71e1269-02e8-45e7-83be-0c1706594a47-kube-api-access-nhdgg" (OuterVolumeSpecName: "kube-api-access-nhdgg") pod "f71e1269-02e8-45e7-83be-0c1706594a47" (UID: "f71e1269-02e8-45e7-83be-0c1706594a47"). InnerVolumeSpecName "kube-api-access-nhdgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:41:46.956875 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:46.956812    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87220fa-3685-4b19-ba33-538f662d6871-kube-api-access-vc7sg" (OuterVolumeSpecName: "kube-api-access-vc7sg") pod "f87220fa-3685-4b19-ba33-538f662d6871" (UID: "f87220fa-3685-4b19-ba33-538f662d6871"). InnerVolumeSpecName "kube-api-access-vc7sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:41:47.055910 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.055882    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc7sg\" (UniqueName: \"kubernetes.io/projected/f87220fa-3685-4b19-ba33-538f662d6871-kube-api-access-vc7sg\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:47.055910 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.055909    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhdgg\" (UniqueName: \"kubernetes.io/projected/f71e1269-02e8-45e7-83be-0c1706594a47-kube-api-access-nhdgg\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:47.056081 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.055918    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87220fa-3685-4b19-ba33-538f662d6871-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:47.056081 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.055927    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71e1269-02e8-45e7-83be-0c1706594a47-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:47.637576 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.637520    2580 generic.go:356] "Generic (PLEG): container finished" podID="f71e1269-02e8-45e7-83be-0c1706594a47" containerID="85be887bf503b577f80a27ba86f37230f9ae9d67b51942a84a4bf07cf3987cf0" exitCode=0
Sep 10 16:41:47.637576 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.637562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7" event={"ID":"f71e1269-02e8-45e7-83be-0c1706594a47","Type":"ContainerDied","Data":"85be887bf503b577f80a27ba86f37230f9ae9d67b51942a84a4bf07cf3987cf0"}
Sep 10 16:41:47.638046 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.637592    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"
Sep 10 16:41:47.638046 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.637650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7" event={"ID":"f71e1269-02e8-45e7-83be-0c1706594a47","Type":"ContainerDied","Data":"97b20e5cc81dd2865b7b2e393408a68650558840cd2ae858831cda266fd73ffc"}
Sep 10 16:41:47.638046 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.637668    2580 scope.go:117] "RemoveContainer" containerID="85be887bf503b577f80a27ba86f37230f9ae9d67b51942a84a4bf07cf3987cf0"
Sep 10 16:41:47.639026 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.638963    2580 generic.go:356] "Generic (PLEG): container finished" podID="f87220fa-3685-4b19-ba33-538f662d6871" containerID="494eaa647581b088d44669e4ec96112ae21beace8d9c681fe5ab80fd5d7a824c" exitCode=0
Sep 10 16:41:47.639026 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.638986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2" event={"ID":"f87220fa-3685-4b19-ba33-538f662d6871","Type":"ContainerDied","Data":"494eaa647581b088d44669e4ec96112ae21beace8d9c681fe5ab80fd5d7a824c"}
Sep 10 16:41:47.639134 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.639021    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"
Sep 10 16:41:47.639134 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.639032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2" event={"ID":"f87220fa-3685-4b19-ba33-538f662d6871","Type":"ContainerDied","Data":"77c284fed0afb3110116c921541528b95a274037a0be3377c9821c0c1dca5f1e"}
Sep 10 16:41:47.646422 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.646405    2580 scope.go:117] "RemoveContainer" containerID="85be887bf503b577f80a27ba86f37230f9ae9d67b51942a84a4bf07cf3987cf0"
Sep 10 16:41:47.646691 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:41:47.646673    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85be887bf503b577f80a27ba86f37230f9ae9d67b51942a84a4bf07cf3987cf0\": container with ID starting with 85be887bf503b577f80a27ba86f37230f9ae9d67b51942a84a4bf07cf3987cf0 not found: ID does not exist" containerID="85be887bf503b577f80a27ba86f37230f9ae9d67b51942a84a4bf07cf3987cf0"
Sep 10 16:41:47.646742 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.646700    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85be887bf503b577f80a27ba86f37230f9ae9d67b51942a84a4bf07cf3987cf0"} err="failed to get container status \"85be887bf503b577f80a27ba86f37230f9ae9d67b51942a84a4bf07cf3987cf0\": rpc error: code = NotFound desc = could not find container \"85be887bf503b577f80a27ba86f37230f9ae9d67b51942a84a4bf07cf3987cf0\": container with ID starting with 85be887bf503b577f80a27ba86f37230f9ae9d67b51942a84a4bf07cf3987cf0 not found: ID does not exist"
Sep 10 16:41:47.646742 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.646717    2580 scope.go:117] "RemoveContainer" containerID="494eaa647581b088d44669e4ec96112ae21beace8d9c681fe5ab80fd5d7a824c"
Sep 10 16:41:47.655247 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.655229    2580 scope.go:117] "RemoveContainer" containerID="494eaa647581b088d44669e4ec96112ae21beace8d9c681fe5ab80fd5d7a824c"
Sep 10 16:41:47.655490 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:41:47.655472    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494eaa647581b088d44669e4ec96112ae21beace8d9c681fe5ab80fd5d7a824c\": container with ID starting with 494eaa647581b088d44669e4ec96112ae21beace8d9c681fe5ab80fd5d7a824c not found: ID does not exist" containerID="494eaa647581b088d44669e4ec96112ae21beace8d9c681fe5ab80fd5d7a824c"
Sep 10 16:41:47.655579 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.655501    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494eaa647581b088d44669e4ec96112ae21beace8d9c681fe5ab80fd5d7a824c"} err="failed to get container status \"494eaa647581b088d44669e4ec96112ae21beace8d9c681fe5ab80fd5d7a824c\": rpc error: code = NotFound desc = could not find container \"494eaa647581b088d44669e4ec96112ae21beace8d9c681fe5ab80fd5d7a824c\": container with ID starting with 494eaa647581b088d44669e4ec96112ae21beace8d9c681fe5ab80fd5d7a824c not found: ID does not exist"
Sep 10 16:41:47.669608 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.669583    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"]
Sep 10 16:41:47.673126 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.673106    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bd1ece-859cbf8c77-b4tm2"]
Sep 10 16:41:47.684441 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.684416    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"]
Sep 10 16:41:47.688074 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:47.688055    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bd1ece-6b4c888bc8-lblb7"]
Sep 10 16:41:48.254471 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:48.254437    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c310823-0d87-4860-8d1f-b84820b7b3a6" path="/var/lib/kubelet/pods/6c310823-0d87-4860-8d1f-b84820b7b3a6/volumes"
Sep 10 16:41:48.254819 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:48.254805    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f71e1269-02e8-45e7-83be-0c1706594a47" path="/var/lib/kubelet/pods/f71e1269-02e8-45e7-83be-0c1706594a47/volumes"
Sep 10 16:41:48.255114 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:48.255103    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f87220fa-3685-4b19-ba33-538f662d6871" path="/var/lib/kubelet/pods/f87220fa-3685-4b19-ba33-538f662d6871/volumes"
Sep 10 16:41:52.306028 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.305994    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7" podUID="52482454-547a-48f9-9cc9-6d73599a0dac" containerName="main" containerID="cri-o://b90c8b5f6ad1aff9ef4f51d5934c2e26b32f2ef43e834dba3fca7800cdb35569" gracePeriod=45
Sep 10 16:41:52.559379 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.559325    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:41:52.607628 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.607603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxkz2\" (UniqueName: \"kubernetes.io/projected/52482454-547a-48f9-9cc9-6d73599a0dac-kube-api-access-bxkz2\") pod \"52482454-547a-48f9-9cc9-6d73599a0dac\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") "
Sep 10 16:41:52.607803 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.607667    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-home\") pod \"52482454-547a-48f9-9cc9-6d73599a0dac\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") "
Sep 10 16:41:52.607803 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.607692    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-tmp-dir\") pod \"52482454-547a-48f9-9cc9-6d73599a0dac\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") "
Sep 10 16:41:52.607803 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.607723    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-model-cache\") pod \"52482454-547a-48f9-9cc9-6d73599a0dac\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") "
Sep 10 16:41:52.607959 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.607807    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-dshm\") pod \"52482454-547a-48f9-9cc9-6d73599a0dac\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") "
Sep 10 16:41:52.607959 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.607863    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52482454-547a-48f9-9cc9-6d73599a0dac-tls-certs\") pod \"52482454-547a-48f9-9cc9-6d73599a0dac\" (UID: \"52482454-547a-48f9-9cc9-6d73599a0dac\") "
Sep 10 16:41:52.607959 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.607897    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-home" (OuterVolumeSpecName: "home") pod "52482454-547a-48f9-9cc9-6d73599a0dac" (UID: "52482454-547a-48f9-9cc9-6d73599a0dac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:41:52.608092 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.607995    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52482454-547a-48f9-9cc9-6d73599a0dac" (UID: "52482454-547a-48f9-9cc9-6d73599a0dac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:41:52.608092 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.608053    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-model-cache" (OuterVolumeSpecName: "model-cache") pod "52482454-547a-48f9-9cc9-6d73599a0dac" (UID: "52482454-547a-48f9-9cc9-6d73599a0dac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:41:52.608176 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.608132    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:52.608176 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.608172    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:52.608292 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.608186    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:52.609779 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.609755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52482454-547a-48f9-9cc9-6d73599a0dac-kube-api-access-bxkz2" (OuterVolumeSpecName: "kube-api-access-bxkz2") pod "52482454-547a-48f9-9cc9-6d73599a0dac" (UID: "52482454-547a-48f9-9cc9-6d73599a0dac"). InnerVolumeSpecName "kube-api-access-bxkz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:41:52.609883 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.609867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52482454-547a-48f9-9cc9-6d73599a0dac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52482454-547a-48f9-9cc9-6d73599a0dac" (UID: "52482454-547a-48f9-9cc9-6d73599a0dac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:41:52.609883 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.609872    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-dshm" (OuterVolumeSpecName: "dshm") pod "52482454-547a-48f9-9cc9-6d73599a0dac" (UID: "52482454-547a-48f9-9cc9-6d73599a0dac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:41:52.665273 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.665245    2580 generic.go:356] "Generic (PLEG): container finished" podID="52482454-547a-48f9-9cc9-6d73599a0dac" containerID="b90c8b5f6ad1aff9ef4f51d5934c2e26b32f2ef43e834dba3fca7800cdb35569" exitCode=0
Sep 10 16:41:52.665465 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.665296    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"
Sep 10 16:41:52.665465 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.665331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7" event={"ID":"52482454-547a-48f9-9cc9-6d73599a0dac","Type":"ContainerDied","Data":"b90c8b5f6ad1aff9ef4f51d5934c2e26b32f2ef43e834dba3fca7800cdb35569"}
Sep 10 16:41:52.665465 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.665384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7" event={"ID":"52482454-547a-48f9-9cc9-6d73599a0dac","Type":"ContainerDied","Data":"74b1f8956bdf771f76351efa847dffe1e19dc0ec39780fe0e0b72a643711d6f1"}
Sep 10 16:41:52.665465 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.665411    2580 scope.go:117] "RemoveContainer" containerID="b90c8b5f6ad1aff9ef4f51d5934c2e26b32f2ef43e834dba3fca7800cdb35569"
Sep 10 16:41:52.674020 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.674004    2580 scope.go:117] "RemoveContainer" containerID="b90c8b5f6ad1aff9ef4f51d5934c2e26b32f2ef43e834dba3fca7800cdb35569"
Sep 10 16:41:52.674289 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:41:52.674270    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90c8b5f6ad1aff9ef4f51d5934c2e26b32f2ef43e834dba3fca7800cdb35569\": container with ID starting with b90c8b5f6ad1aff9ef4f51d5934c2e26b32f2ef43e834dba3fca7800cdb35569 not found: ID does not exist" containerID="b90c8b5f6ad1aff9ef4f51d5934c2e26b32f2ef43e834dba3fca7800cdb35569"
Sep 10 16:41:52.674355 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.674299    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90c8b5f6ad1aff9ef4f51d5934c2e26b32f2ef43e834dba3fca7800cdb35569"} err="failed to get container status \"b90c8b5f6ad1aff9ef4f51d5934c2e26b32f2ef43e834dba3fca7800cdb35569\": rpc error: code = NotFound desc = could not find container \"b90c8b5f6ad1aff9ef4f51d5934c2e26b32f2ef43e834dba3fca7800cdb35569\": container with ID starting with b90c8b5f6ad1aff9ef4f51d5934c2e26b32f2ef43e834dba3fca7800cdb35569 not found: ID does not exist"
Sep 10 16:41:52.687365 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.687319    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"]
Sep 10 16:41:52.689590 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.689568    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1925d8/rate-test-model-1925d8-kserve-64b848ccbc-d76t7"]
Sep 10 16:41:52.708877 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.708858    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52482454-547a-48f9-9cc9-6d73599a0dac-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:52.708877 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.708878    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52482454-547a-48f9-9cc9-6d73599a0dac-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:52.708985 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:52.708891    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxkz2\" (UniqueName: \"kubernetes.io/projected/52482454-547a-48f9-9cc9-6d73599a0dac-kube-api-access-bxkz2\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:53.256375 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.256316    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd" podUID="82ee96c5-2189-44f5-bbe9-eca0b013a28f" containerName="main" containerID="cri-o://7d2b93a551b8b1ccabeeba989111af3f2d2d12a990cff3b2fa3205556e0a90e4" gracePeriod=45
Sep 10 16:41:53.419486 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.419445    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd" podUID="82ee96c5-2189-44f5-bbe9-eca0b013a28f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 10 16:41:53.495257 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.495235    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:41:53.614500 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.614473    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-model-cache\") pod \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") "
Sep 10 16:41:53.614660 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.614508    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-tmp-dir\") pod \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") "
Sep 10 16:41:53.614660 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.614569    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjx5k\" (UniqueName: \"kubernetes.io/projected/82ee96c5-2189-44f5-bbe9-eca0b013a28f-kube-api-access-wjx5k\") pod \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") "
Sep 10 16:41:53.614660 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.614595    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-home\") pod \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") "
Sep 10 16:41:53.614820 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.614664    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82ee96c5-2189-44f5-bbe9-eca0b013a28f-tls-certs\") pod \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") "
Sep 10 16:41:53.614820 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.614708    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-dshm\") pod \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\" (UID: \"82ee96c5-2189-44f5-bbe9-eca0b013a28f\") "
Sep 10 16:41:53.614820 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.614796    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-model-cache" (OuterVolumeSpecName: "model-cache") pod "82ee96c5-2189-44f5-bbe9-eca0b013a28f" (UID: "82ee96c5-2189-44f5-bbe9-eca0b013a28f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:41:53.614980 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.614825    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82ee96c5-2189-44f5-bbe9-eca0b013a28f" (UID: "82ee96c5-2189-44f5-bbe9-eca0b013a28f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:41:53.614980 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.614935    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-home" (OuterVolumeSpecName: "home") pod "82ee96c5-2189-44f5-bbe9-eca0b013a28f" (UID: "82ee96c5-2189-44f5-bbe9-eca0b013a28f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:41:53.614980 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.614960    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:53.614980 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.614978    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:53.616596 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.616565    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ee96c5-2189-44f5-bbe9-eca0b013a28f-kube-api-access-wjx5k" (OuterVolumeSpecName: "kube-api-access-wjx5k") pod "82ee96c5-2189-44f5-bbe9-eca0b013a28f" (UID: "82ee96c5-2189-44f5-bbe9-eca0b013a28f"). InnerVolumeSpecName "kube-api-access-wjx5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:41:53.616709 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.616599    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ee96c5-2189-44f5-bbe9-eca0b013a28f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82ee96c5-2189-44f5-bbe9-eca0b013a28f" (UID: "82ee96c5-2189-44f5-bbe9-eca0b013a28f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:41:53.616709 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.616667    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-dshm" (OuterVolumeSpecName: "dshm") pod "82ee96c5-2189-44f5-bbe9-eca0b013a28f" (UID: "82ee96c5-2189-44f5-bbe9-eca0b013a28f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:41:53.671753 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.671727    2580 generic.go:356] "Generic (PLEG): container finished" podID="82ee96c5-2189-44f5-bbe9-eca0b013a28f" containerID="7d2b93a551b8b1ccabeeba989111af3f2d2d12a990cff3b2fa3205556e0a90e4" exitCode=0
Sep 10 16:41:53.671853 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.671769    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"
Sep 10 16:41:53.671853 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.671788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd" event={"ID":"82ee96c5-2189-44f5-bbe9-eca0b013a28f","Type":"ContainerDied","Data":"7d2b93a551b8b1ccabeeba989111af3f2d2d12a990cff3b2fa3205556e0a90e4"}
Sep 10 16:41:53.671853 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.671821    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd" event={"ID":"82ee96c5-2189-44f5-bbe9-eca0b013a28f","Type":"ContainerDied","Data":"dec2b46ab1b5cdcbfcffee3b2b5dce5f35c63ffb4546039d2f856a0f824df1ec"}
Sep 10 16:41:53.671853 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.671844    2580 scope.go:117] "RemoveContainer" containerID="7d2b93a551b8b1ccabeeba989111af3f2d2d12a990cff3b2fa3205556e0a90e4"
Sep 10 16:41:53.680003 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.679979    2580 scope.go:117] "RemoveContainer" containerID="7d2b93a551b8b1ccabeeba989111af3f2d2d12a990cff3b2fa3205556e0a90e4"
Sep 10 16:41:53.680265 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:41:53.680245    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d2b93a551b8b1ccabeeba989111af3f2d2d12a990cff3b2fa3205556e0a90e4\": container with ID starting with 7d2b93a551b8b1ccabeeba989111af3f2d2d12a990cff3b2fa3205556e0a90e4 not found: ID does not exist" containerID="7d2b93a551b8b1ccabeeba989111af3f2d2d12a990cff3b2fa3205556e0a90e4"
Sep 10 16:41:53.680327 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.680272    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d2b93a551b8b1ccabeeba989111af3f2d2d12a990cff3b2fa3205556e0a90e4"} err="failed to get container status \"7d2b93a551b8b1ccabeeba989111af3f2d2d12a990cff3b2fa3205556e0a90e4\": rpc error: code = NotFound desc = could not find container \"7d2b93a551b8b1ccabeeba989111af3f2d2d12a990cff3b2fa3205556e0a90e4\": container with ID starting with 7d2b93a551b8b1ccabeeba989111af3f2d2d12a990cff3b2fa3205556e0a90e4 not found: ID does not exist"
Sep 10 16:41:53.692635 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.692613    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"]
Sep 10 16:41:53.695437 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.695417    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ccbd4/rate-test-model-3ccbd4-kserve-6f8c785b87-x4phd"]
Sep 10 16:41:53.716092 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.716073    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82ee96c5-2189-44f5-bbe9-eca0b013a28f-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:53.716092 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.716091    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:53.716178 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.716099    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjx5k\" (UniqueName: \"kubernetes.io/projected/82ee96c5-2189-44f5-bbe9-eca0b013a28f-kube-api-access-wjx5k\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:53.716178 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:53.716108    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82ee96c5-2189-44f5-bbe9-eca0b013a28f-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:54.254584 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.254552    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52482454-547a-48f9-9cc9-6d73599a0dac" path="/var/lib/kubelet/pods/52482454-547a-48f9-9cc9-6d73599a0dac/volumes"
Sep 10 16:41:54.254956 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.254943    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ee96c5-2189-44f5-bbe9-eca0b013a28f" path="/var/lib/kubelet/pods/82ee96c5-2189-44f5-bbe9-eca0b013a28f/volumes"
Sep 10 16:41:54.674780 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.674750    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng"]
Sep 10 16:41:54.675137 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675075    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f87220fa-3685-4b19-ba33-538f662d6871" containerName="payload-processing"
Sep 10 16:41:54.675137 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675088    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f87220fa-3685-4b19-ba33-538f662d6871" containerName="payload-processing"
Sep 10 16:41:54.675137 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675100    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ee96c5-2189-44f5-bbe9-eca0b013a28f" containerName="main"
Sep 10 16:41:54.675137 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675105    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ee96c5-2189-44f5-bbe9-eca0b013a28f" containerName="main"
Sep 10 16:41:54.675137 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675118    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52482454-547a-48f9-9cc9-6d73599a0dac" containerName="main"
Sep 10 16:41:54.675137 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675127    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="52482454-547a-48f9-9cc9-6d73599a0dac" containerName="main"
Sep 10 16:41:54.675137 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675133    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c310823-0d87-4860-8d1f-b84820b7b3a6" containerName="revoke-keys"
Sep 10 16:41:54.675137 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675139    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c310823-0d87-4860-8d1f-b84820b7b3a6" containerName="revoke-keys"
Sep 10 16:41:54.675415 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675150    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71e1269-02e8-45e7-83be-0c1706594a47" containerName="payload-pre-processing"
Sep 10 16:41:54.675415 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675156    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71e1269-02e8-45e7-83be-0c1706594a47" containerName="payload-pre-processing"
Sep 10 16:41:54.675415 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675209    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ee96c5-2189-44f5-bbe9-eca0b013a28f" containerName="main"
Sep 10 16:41:54.675415 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675218    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f87220fa-3685-4b19-ba33-538f662d6871" containerName="payload-processing"
Sep 10 16:41:54.675415 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675225    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f71e1269-02e8-45e7-83be-0c1706594a47" containerName="payload-pre-processing"
Sep 10 16:41:54.675415 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675230    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="52482454-547a-48f9-9cc9-6d73599a0dac" containerName="main"
Sep 10 16:41:54.675415 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.675238    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c310823-0d87-4860-8d1f-b84820b7b3a6" containerName="revoke-keys"
Sep 10 16:41:54.679689 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.679668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng"
Sep 10 16:41:54.682327 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.682308    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:41:54.682420 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.682401    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\""
Sep 10 16:41:54.687359 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.687335    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng"]
Sep 10 16:41:54.724195 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.724163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203cdb57-0491-4983-b7cd-c1b882dc4b81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng\" (UID: \"203cdb57-0491-4983-b7cd-c1b882dc4b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng"
Sep 10 16:41:54.825260 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.825226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203cdb57-0491-4983-b7cd-c1b882dc4b81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng\" (UID: \"203cdb57-0491-4983-b7cd-c1b882dc4b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng"
Sep 10 16:41:54.825821 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.825803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203cdb57-0491-4983-b7cd-c1b882dc4b81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng\" (UID: \"203cdb57-0491-4983-b7cd-c1b882dc4b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng"
Sep 10 16:41:54.990440 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:54.990383    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng"
Sep 10 16:41:55.260637 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:55.260594    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng"]
Sep 10 16:41:55.264509 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:41:55.264483    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203cdb57_0491_4983_b7cd_c1b882dc4b81.slice/crio-bace868ddceb6d77ddf6774a5e1554509db1a669aae1f7eff07dd4646c0c8631 WatchSource:0}: Error finding container bace868ddceb6d77ddf6774a5e1554509db1a669aae1f7eff07dd4646c0c8631: Status 404 returned error can't find the container with id bace868ddceb6d77ddf6774a5e1554509db1a669aae1f7eff07dd4646c0c8631
Sep 10 16:41:55.684065 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:55.684032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng" event={"ID":"203cdb57-0491-4983-b7cd-c1b882dc4b81","Type":"ContainerStarted","Data":"8a01976a85fbe20ba4c84e7dae8551caaa8fc6b40c46b24753386039954a2622"}
Sep 10 16:41:55.684380 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:55.684083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng" event={"ID":"203cdb57-0491-4983-b7cd-c1b882dc4b81","Type":"ContainerStarted","Data":"bace868ddceb6d77ddf6774a5e1554509db1a669aae1f7eff07dd4646c0c8631"}
Sep 10 16:41:55.705329 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:55.705289    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng" podStartSLOduration=1.7052750589999999 podStartE2EDuration="1.705275059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:41:55.70509809 +0000 UTC m=+1473.906140504" watchObservedRunningTime="2026-09-10 16:41:55.705275059 +0000 UTC m=+1473.906317470"
Sep 10 16:41:56.690162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:56.690122    2580 generic.go:356] "Generic (PLEG): container finished" podID="203cdb57-0491-4983-b7cd-c1b882dc4b81" containerID="8a01976a85fbe20ba4c84e7dae8551caaa8fc6b40c46b24753386039954a2622" exitCode=0
Sep 10 16:41:56.690518 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:56.690210    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng" event={"ID":"203cdb57-0491-4983-b7cd-c1b882dc4b81","Type":"ContainerDied","Data":"8a01976a85fbe20ba4c84e7dae8551caaa8fc6b40c46b24753386039954a2622"}
Sep 10 16:41:57.826129 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:57.826104    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng"
Sep 10 16:41:57.950913 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:57.950833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203cdb57-0491-4983-b7cd-c1b882dc4b81-maas-api-service-ca\") pod \"203cdb57-0491-4983-b7cd-c1b882dc4b81\" (UID: \"203cdb57-0491-4983-b7cd-c1b882dc4b81\") "
Sep 10 16:41:57.951131 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:57.951109    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/203cdb57-0491-4983-b7cd-c1b882dc4b81-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "203cdb57-0491-4983-b7cd-c1b882dc4b81" (UID: "203cdb57-0491-4983-b7cd-c1b882dc4b81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:41:58.051789 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:58.051762    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203cdb57-0491-4983-b7cd-c1b882dc4b81-maas-api-service-ca\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:41:58.700962 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:58.700873    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng" event={"ID":"203cdb57-0491-4983-b7cd-c1b882dc4b81","Type":"ContainerDied","Data":"bace868ddceb6d77ddf6774a5e1554509db1a669aae1f7eff07dd4646c0c8631"}
Sep 10 16:41:58.700962 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:58.700920    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bace868ddceb6d77ddf6774a5e1554509db1a669aae1f7eff07dd4646c0c8631"
Sep 10 16:41:58.700962 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:41:58.700883    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng"
Sep 10 16:42:02.801273 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.801239    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"]
Sep 10 16:42:02.801666 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.801620    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="203cdb57-0491-4983-b7cd-c1b882dc4b81" containerName="revoke-keys"
Sep 10 16:42:02.801666 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.801632    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="203cdb57-0491-4983-b7cd-c1b882dc4b81" containerName="revoke-keys"
Sep 10 16:42:02.801746 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.801693    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="203cdb57-0491-4983-b7cd-c1b882dc4b81" containerName="revoke-keys"
Sep 10 16:42:02.804246 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.804227    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.807104 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.807084    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 16:42:02.807217 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.807182    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-86d230-openshift-default-dockercfg-xkb2d\""
Sep 10 16:42:02.817774 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.817752    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"]
Sep 10 16:42:02.889920 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.889879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txxcg\" (UniqueName: \"kubernetes.io/projected/92367ce3-7f8b-4a38-a92e-ce005b81b74b-kube-api-access-txxcg\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.890106 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.889928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-token\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.890106 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.889956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-envoy\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.890106 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.889976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-podinfo\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.890106 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.890002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-workload-certs\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.890106 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.890018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istiod-ca-cert\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.890106 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.890086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-data\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.890386 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.890124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-workload-socket\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.890386 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.890161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-credential-socket\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.990945 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.990911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-data\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.990945 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.990947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-workload-socket\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.991171 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.990973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-credential-socket\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.991171 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.991010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxcg\" (UniqueName: \"kubernetes.io/projected/92367ce3-7f8b-4a38-a92e-ce005b81b74b-kube-api-access-txxcg\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.991171 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.991048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-token\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.991324 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.991204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-envoy\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.991324 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.991260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-podinfo\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.991324 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.991315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-workload-certs\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.991481 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.991339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istiod-ca-cert\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.991481 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.991393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-workload-socket\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.991481 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.991337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-data\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.991481 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.991440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-credential-socket\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.991712 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.991687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-workload-certs\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.991917 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.991899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istiod-ca-cert\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.993483 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.993463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-envoy\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:02.993728 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:02.993708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-podinfo\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:03.000851 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:03.000822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-token\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:03.004185 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:03.004165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxcg\" (UniqueName: \"kubernetes.io/projected/92367ce3-7f8b-4a38-a92e-ce005b81b74b-kube-api-access-txxcg\") pod \"e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") " pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:03.114577 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:03.114459    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:03.392163 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:03.392139    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"]
Sep 10 16:42:03.394214 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:42:03.394188    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92367ce3_7f8b_4a38_a92e_ce005b81b74b.slice/crio-508e70e948ced98f8a80c84d74f592eb00df7112d2b9af56c1f1c60c936d56bb WatchSource:0}: Error finding container 508e70e948ced98f8a80c84d74f592eb00df7112d2b9af56c1f1c60c936d56bb: Status 404 returned error can't find the container with id 508e70e948ced98f8a80c84d74f592eb00df7112d2b9af56c1f1c60c936d56bb
Sep 10 16:42:03.396272 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:03.396256    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 16:42:03.729562 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:03.729462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7" event={"ID":"92367ce3-7f8b-4a38-a92e-ce005b81b74b","Type":"ContainerStarted","Data":"508e70e948ced98f8a80c84d74f592eb00df7112d2b9af56c1f1c60c936d56bb"}
Sep 10 16:42:05.860616 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:05.860576    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng"]
Sep 10 16:42:05.863892 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:05.863866    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-035bd5-pr4ng"]
Sep 10 16:42:05.908957 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:05.908920    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 16:42:05.909075 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:05.909002    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 16:42:05.909075 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:05.909046    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 16:42:06.254662 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:06.254574    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="203cdb57-0491-4983-b7cd-c1b882dc4b81" path="/var/lib/kubelet/pods/203cdb57-0491-4983-b7cd-c1b882dc4b81/volumes"
Sep 10 16:42:06.746238 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:06.746202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7" event={"ID":"92367ce3-7f8b-4a38-a92e-ce005b81b74b","Type":"ContainerStarted","Data":"36836c20829f3cd16cc3409fc2a40a10bb370f5decf968bfc758e97e8d6eea09"}
Sep 10 16:42:06.768426 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:06.768375    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7" podStartSLOduration=2.25615372 podStartE2EDuration="4.76836035s" totalImagesPullingTime="2.51220663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:42:06.767132175 +0000 UTC m=+1484.968174588" watchObservedRunningTime="2026-09-10 16:42:06.76836035 +0000 UTC m=+1484.969402762"
Sep 10 16:42:07.115175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:07.115145    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:07.120375 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:07.120350    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:07.751511 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:07.751483    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:07.752598 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:07.752575    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:10.628106 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:10.628066    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"]
Sep 10 16:42:10.632013 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:10.631993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"
Sep 10 16:42:10.634372 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:10.634350    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-86d230\""
Sep 10 16:42:10.634480 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:10.634408    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-86d230-dockercfg-lxwlg\""
Sep 10 16:42:10.643582 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:10.643560    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"]
Sep 10 16:42:10.763686 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:10.763640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6894ba38-4323-4453-b1dc-bd90334b6c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n\" (UID: \"6894ba38-4323-4453-b1dc-bd90334b6c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"
Sep 10 16:42:10.763848 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:10.763736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkm5m\" (UniqueName: \"kubernetes.io/projected/6894ba38-4323-4453-b1dc-bd90334b6c08-kube-api-access-jkm5m\") pod \"payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n\" (UID: \"6894ba38-4323-4453-b1dc-bd90334b6c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"
Sep 10 16:42:10.864495 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:10.864463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6894ba38-4323-4453-b1dc-bd90334b6c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n\" (UID: \"6894ba38-4323-4453-b1dc-bd90334b6c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"
Sep 10 16:42:10.864658 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:10.864517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkm5m\" (UniqueName: \"kubernetes.io/projected/6894ba38-4323-4453-b1dc-bd90334b6c08-kube-api-access-jkm5m\") pod \"payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n\" (UID: \"6894ba38-4323-4453-b1dc-bd90334b6c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"
Sep 10 16:42:10.865224 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:10.865200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6894ba38-4323-4453-b1dc-bd90334b6c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n\" (UID: \"6894ba38-4323-4453-b1dc-bd90334b6c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"
Sep 10 16:42:10.873208 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:10.873188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkm5m\" (UniqueName: \"kubernetes.io/projected/6894ba38-4323-4453-b1dc-bd90334b6c08-kube-api-access-jkm5m\") pod \"payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n\" (UID: \"6894ba38-4323-4453-b1dc-bd90334b6c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"
Sep 10 16:42:10.941836 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:10.941776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"
Sep 10 16:42:11.215862 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:11.215838    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"]
Sep 10 16:42:11.218494 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:42:11.218472    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6894ba38_4323_4453_b1dc_bd90334b6c08.slice/crio-b8e7d9588778294f1cb7e7a6ad519420377c78617b8b323a145d2121cea2c55e WatchSource:0}: Error finding container b8e7d9588778294f1cb7e7a6ad519420377c78617b8b323a145d2121cea2c55e: Status 404 returned error can't find the container with id b8e7d9588778294f1cb7e7a6ad519420377c78617b8b323a145d2121cea2c55e
Sep 10 16:42:11.771343 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:11.771311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n" event={"ID":"6894ba38-4323-4453-b1dc-bd90334b6c08","Type":"ContainerStarted","Data":"65eb7542b37f747b28ef223b94c7128203ebec9fef2b8c4f5faad8833b734893"}
Sep 10 16:42:11.771826 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:11.771357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n" event={"ID":"6894ba38-4323-4453-b1dc-bd90334b6c08","Type":"ContainerStarted","Data":"b8e7d9588778294f1cb7e7a6ad519420377c78617b8b323a145d2121cea2c55e"}
Sep 10 16:42:11.771826 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:11.771405    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"
Sep 10 16:42:11.790849 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:11.790805    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n" podStartSLOduration=1.790792543 podStartE2EDuration="1.790792543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:42:11.789243316 +0000 UTC m=+1489.990285727" watchObservedRunningTime="2026-09-10 16:42:11.790792543 +0000 UTC m=+1489.991834972"
Sep 10 16:42:22.777862 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:22.777833    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"
Sep 10 16:42:27.854906 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:27.854872    2580 generic.go:356] "Generic (PLEG): container finished" podID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerID="36836c20829f3cd16cc3409fc2a40a10bb370f5decf968bfc758e97e8d6eea09" exitCode=0
Sep 10 16:42:27.855402 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:27.854935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7" event={"ID":"92367ce3-7f8b-4a38-a92e-ce005b81b74b","Type":"ContainerDied","Data":"36836c20829f3cd16cc3409fc2a40a10bb370f5decf968bfc758e97e8d6eea09"}
Sep 10 16:42:27.855402 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:27.855387    2580 scope.go:117] "RemoveContainer" containerID="36836c20829f3cd16cc3409fc2a40a10bb370f5decf968bfc758e97e8d6eea09"
Sep 10 16:42:27.856526 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:27.856496    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 16:42:27.856648 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:27.856607    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 16:42:27.856713 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:27.856660    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 16:42:28.861426 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:28.861395    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7" event={"ID":"92367ce3-7f8b-4a38-a92e-ce005b81b74b","Type":"ContainerStarted","Data":"fc3b5cb8bdefb6066d899c71ff3fde650fa4af9263f8b2108f88b20d3e748aa2"}
Sep 10 16:42:29.115765 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:29.115671    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:29.116100 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:29.116075    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 10 16:42:29.116247 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:29.116223    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7" podUID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 10 16:42:30.115849 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:30.115801    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 10 16:42:30.116356 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:30.115874    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7" podUID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 10 16:42:31.115545 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:31.115483    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 10 16:42:31.115766 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:31.115589    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7" podUID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 10 16:42:32.115263 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:32.115224    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 10 16:42:32.115870 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:32.115836    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7" podUID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 10 16:42:33.114898 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:33.114855    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 10 16:42:33.115100 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:33.114920    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7" podUID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 10 16:42:34.115679 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:34.115635    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 10 16:42:34.116104 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:34.115706    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7" podUID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 10 16:42:35.119196 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:35.119165    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:35.119641 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:35.119563    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:35.120345 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:35.120328    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:42:38.731980 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:38.731944    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mr4zw_d6605008-787d-480d-b24b-5e343dfe6d4d/payload-processing/0.log"
Sep 10 16:42:57.653564 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.653482    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"]
Sep 10 16:42:57.656782 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.656765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.659319 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.659296    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-95d261\"/\"default-dockercfg-db78m\""
Sep 10 16:42:57.659440 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.659296    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-95d261\"/\"openshift-service-ca.crt\""
Sep 10 16:42:57.659836 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.659820    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-95d261\"/\"ipp-route-95d261-kserve-self-signed-certs\""
Sep 10 16:42:57.660472 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.660457    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-95d261\"/\"kube-root-ca.crt\""
Sep 10 16:42:57.670644 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.670621    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"]
Sep 10 16:42:57.732114 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.732080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-home\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.732306 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.732137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-tmp-dir\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.732306 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.732190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-model-cache\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.732306 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.732240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-dshm\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.732306 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.732268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-tls-certs\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.732470 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.732317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd5t6\" (UniqueName: \"kubernetes.io/projected/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-kube-api-access-hd5t6\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.833398 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.833359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-home\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.833569 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.833414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-tmp-dir\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.833569 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.833435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-model-cache\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.833569 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.833560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-dshm\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.833744 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.833599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-tls-certs\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.833744 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.833635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd5t6\" (UniqueName: \"kubernetes.io/projected/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-kube-api-access-hd5t6\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.833850 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.833801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-model-cache\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.833908 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.833882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-tmp-dir\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.833908 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.833898    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-home\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.835915 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.835887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-dshm\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.836018 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.835999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-tls-certs\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.842043 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.842019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd5t6\" (UniqueName: \"kubernetes.io/projected/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-kube-api-access-hd5t6\") pod \"ipp-route-95d261-kserve-6449846b7f-f54sz\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") " pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:57.966204 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:57.966133    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:42:58.240112 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:58.240084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"]
Sep 10 16:42:58.242074 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:42:58.242048    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e9a46f0_7b2d_4200_a98e_4c9f80c30828.slice/crio-2a90c0b2ded14198ec5edf39db8966601a3ef471c3186b7577e53e75162708fe WatchSource:0}: Error finding container 2a90c0b2ded14198ec5edf39db8966601a3ef471c3186b7577e53e75162708fe: Status 404 returned error can't find the container with id 2a90c0b2ded14198ec5edf39db8966601a3ef471c3186b7577e53e75162708fe
Sep 10 16:42:59.001169 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:59.001135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz" event={"ID":"1e9a46f0-7b2d-4200-a98e-4c9f80c30828","Type":"ContainerStarted","Data":"a5b55cd8be8c14658db5744163a02cd3777b0dab8ba63a2a262b6348041f6bc4"}
Sep 10 16:42:59.001169 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:59.001179    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz" event={"ID":"1e9a46f0-7b2d-4200-a98e-4c9f80c30828","Type":"ContainerStarted","Data":"2a90c0b2ded14198ec5edf39db8966601a3ef471c3186b7577e53e75162708fe"}
Sep 10 16:42:59.020791 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:42:59.020748    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz" podStartSLOduration=2.020734087 podStartE2EDuration="2.020734087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:42:59.019298952 +0000 UTC m=+1537.220341365" watchObservedRunningTime="2026-09-10 16:42:59.020734087 +0000 UTC m=+1537.221776548"
Sep 10 16:43:07.967192 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:07.967155    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:43:07.967192 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:07.967199    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:43:07.979595 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:07.979572    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:43:08.055479 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:08.055444    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:43:30.020140 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:30.020108    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mr4zw_d6605008-787d-480d-b24b-5e343dfe6d4d/payload-processing/0.log"
Sep 10 16:43:37.613036 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:37.613000    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"]
Sep 10 16:43:37.616604 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:37.616589    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"
Sep 10 16:43:37.619160 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:37.619141    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-f9d55d\""
Sep 10 16:43:37.624665 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:37.624643    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"]
Sep 10 16:43:37.655586 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:37.655556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83fbb6be-6fec-4100-ae0c-fecf7b5ab708-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x\" (UID: \"83fbb6be-6fec-4100-ae0c-fecf7b5ab708\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"
Sep 10 16:43:37.655705 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:37.655598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xskcz\" (UniqueName: \"kubernetes.io/projected/83fbb6be-6fec-4100-ae0c-fecf7b5ab708-kube-api-access-xskcz\") pod \"payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x\" (UID: \"83fbb6be-6fec-4100-ae0c-fecf7b5ab708\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"
Sep 10 16:43:37.756989 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:37.756957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83fbb6be-6fec-4100-ae0c-fecf7b5ab708-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x\" (UID: \"83fbb6be-6fec-4100-ae0c-fecf7b5ab708\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"
Sep 10 16:43:37.757149 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:37.756994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xskcz\" (UniqueName: \"kubernetes.io/projected/83fbb6be-6fec-4100-ae0c-fecf7b5ab708-kube-api-access-xskcz\") pod \"payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x\" (UID: \"83fbb6be-6fec-4100-ae0c-fecf7b5ab708\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"
Sep 10 16:43:37.757697 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:37.757677    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83fbb6be-6fec-4100-ae0c-fecf7b5ab708-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x\" (UID: \"83fbb6be-6fec-4100-ae0c-fecf7b5ab708\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"
Sep 10 16:43:37.765985 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:37.765960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xskcz\" (UniqueName: \"kubernetes.io/projected/83fbb6be-6fec-4100-ae0c-fecf7b5ab708-kube-api-access-xskcz\") pod \"payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x\" (UID: \"83fbb6be-6fec-4100-ae0c-fecf7b5ab708\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"
Sep 10 16:43:37.927378 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:37.927295    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"
Sep 10 16:43:38.211685 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:38.211660    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"]
Sep 10 16:43:38.213812 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:43:38.213782    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fbb6be_6fec_4100_ae0c_fecf7b5ab708.slice/crio-f9013bad6d0f4b3d1a7ebeae2a63457d317a60ec930f2d7962fb12c7c1055f54 WatchSource:0}: Error finding container f9013bad6d0f4b3d1a7ebeae2a63457d317a60ec930f2d7962fb12c7c1055f54: Status 404 returned error can't find the container with id f9013bad6d0f4b3d1a7ebeae2a63457d317a60ec930f2d7962fb12c7c1055f54
Sep 10 16:43:39.192509 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:39.192472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x" event={"ID":"83fbb6be-6fec-4100-ae0c-fecf7b5ab708","Type":"ContainerStarted","Data":"98b328e3ed142cc42dae121eaf268eb6efba09f462bfd4d2d31f5c6809e3efe6"}
Sep 10 16:43:39.193005 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:39.192549    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x" event={"ID":"83fbb6be-6fec-4100-ae0c-fecf7b5ab708","Type":"ContainerStarted","Data":"f9013bad6d0f4b3d1a7ebeae2a63457d317a60ec930f2d7962fb12c7c1055f54"}
Sep 10 16:43:39.193005 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:39.192610    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"
Sep 10 16:43:39.211474 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:39.211420    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x" podStartSLOduration=2.211404769 podStartE2EDuration="2.211404769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:43:39.210271529 +0000 UTC m=+1577.411313940" watchObservedRunningTime="2026-09-10 16:43:39.211404769 +0000 UTC m=+1577.412447179"
Sep 10 16:43:50.199613 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:43:50.199582    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"
Sep 10 16:44:04.289111 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:04.289065    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb"]
Sep 10 16:44:04.292497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:04.292473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb"
Sep 10 16:44:04.294924 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:04.294898    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:44:04.295049 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:04.294909    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\""
Sep 10 16:44:04.300596 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:04.300574    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb"]
Sep 10 16:44:04.396279 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:04.396234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5671ec2a-f178-497c-9edd-dd4e11e08c68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb\" (UID: \"5671ec2a-f178-497c-9edd-dd4e11e08c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb"
Sep 10 16:44:04.497813 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:04.497762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5671ec2a-f178-497c-9edd-dd4e11e08c68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb\" (UID: \"5671ec2a-f178-497c-9edd-dd4e11e08c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb"
Sep 10 16:44:04.498385 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:04.498365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5671ec2a-f178-497c-9edd-dd4e11e08c68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb\" (UID: \"5671ec2a-f178-497c-9edd-dd4e11e08c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb"
Sep 10 16:44:04.604516 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:04.604417    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb"
Sep 10 16:44:04.878278 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:04.878246    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb"]
Sep 10 16:44:04.881065 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:44:04.881039    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5671ec2a_f178_497c_9edd_dd4e11e08c68.slice/crio-378d00818270d053e5ea1e76525c02aa6d51184462515ba711910eeafd0719b1 WatchSource:0}: Error finding container 378d00818270d053e5ea1e76525c02aa6d51184462515ba711910eeafd0719b1: Status 404 returned error can't find the container with id 378d00818270d053e5ea1e76525c02aa6d51184462515ba711910eeafd0719b1
Sep 10 16:44:05.312338 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:05.312302    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb" event={"ID":"5671ec2a-f178-497c-9edd-dd4e11e08c68","Type":"ContainerStarted","Data":"9d7bf12eb02c269c842e80b502b87f43dcb342df1ba503c5b12a2e076cade601"}
Sep 10 16:44:05.312770 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:05.312358    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb" event={"ID":"5671ec2a-f178-497c-9edd-dd4e11e08c68","Type":"ContainerStarted","Data":"378d00818270d053e5ea1e76525c02aa6d51184462515ba711910eeafd0719b1"}
Sep 10 16:44:05.329759 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:05.329710    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb" podStartSLOduration=1.329694267 podStartE2EDuration="1.329694267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:44:05.328925316 +0000 UTC m=+1603.529967736" watchObservedRunningTime="2026-09-10 16:44:05.329694267 +0000 UTC m=+1603.530736698"
Sep 10 16:44:06.318998 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:06.318967    2580 generic.go:356] "Generic (PLEG): container finished" podID="5671ec2a-f178-497c-9edd-dd4e11e08c68" containerID="9d7bf12eb02c269c842e80b502b87f43dcb342df1ba503c5b12a2e076cade601" exitCode=0
Sep 10 16:44:06.319381 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:06.319057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb" event={"ID":"5671ec2a-f178-497c-9edd-dd4e11e08c68","Type":"ContainerDied","Data":"9d7bf12eb02c269c842e80b502b87f43dcb342df1ba503c5b12a2e076cade601"}
Sep 10 16:44:07.459305 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:07.459274    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb"
Sep 10 16:44:07.627925 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:07.627829    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5671ec2a-f178-497c-9edd-dd4e11e08c68-maas-api-service-ca\") pod \"5671ec2a-f178-497c-9edd-dd4e11e08c68\" (UID: \"5671ec2a-f178-497c-9edd-dd4e11e08c68\") "
Sep 10 16:44:07.628202 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:07.628178    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5671ec2a-f178-497c-9edd-dd4e11e08c68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5671ec2a-f178-497c-9edd-dd4e11e08c68" (UID: "5671ec2a-f178-497c-9edd-dd4e11e08c68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:44:07.728764 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:07.728724    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5671ec2a-f178-497c-9edd-dd4e11e08c68-maas-api-service-ca\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:44:08.329710 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:08.329679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb" event={"ID":"5671ec2a-f178-497c-9edd-dd4e11e08c68","Type":"ContainerDied","Data":"378d00818270d053e5ea1e76525c02aa6d51184462515ba711910eeafd0719b1"}
Sep 10 16:44:08.329857 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:08.329715    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb"
Sep 10 16:44:08.329857 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:08.329728    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="378d00818270d053e5ea1e76525c02aa6d51184462515ba711910eeafd0719b1"
Sep 10 16:44:15.558113 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:15.558013    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"]
Sep 10 16:44:15.558765 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:15.558281    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x" podUID="83fbb6be-6fec-4100-ae0c-fecf7b5ab708" containerName="payload-pre-processing" containerID="cri-o://98b328e3ed142cc42dae121eaf268eb6efba09f462bfd4d2d31f5c6809e3efe6" gracePeriod=30
Sep 10 16:44:15.719754 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:15.719720    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb"]
Sep 10 16:44:15.723872 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:15.723844    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f9d55d-tjrrb"]
Sep 10 16:44:15.805642 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:15.805615    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"
Sep 10 16:44:15.905649 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:15.905563    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83fbb6be-6fec-4100-ae0c-fecf7b5ab708-plugins-config-volume\") pod \"83fbb6be-6fec-4100-ae0c-fecf7b5ab708\" (UID: \"83fbb6be-6fec-4100-ae0c-fecf7b5ab708\") "
Sep 10 16:44:15.905796 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:15.905650    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xskcz\" (UniqueName: \"kubernetes.io/projected/83fbb6be-6fec-4100-ae0c-fecf7b5ab708-kube-api-access-xskcz\") pod \"83fbb6be-6fec-4100-ae0c-fecf7b5ab708\" (UID: \"83fbb6be-6fec-4100-ae0c-fecf7b5ab708\") "
Sep 10 16:44:15.905949 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:15.905929    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fbb6be-6fec-4100-ae0c-fecf7b5ab708-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "83fbb6be-6fec-4100-ae0c-fecf7b5ab708" (UID: "83fbb6be-6fec-4100-ae0c-fecf7b5ab708"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:44:15.907730 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:15.907701    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fbb6be-6fec-4100-ae0c-fecf7b5ab708-kube-api-access-xskcz" (OuterVolumeSpecName: "kube-api-access-xskcz") pod "83fbb6be-6fec-4100-ae0c-fecf7b5ab708" (UID: "83fbb6be-6fec-4100-ae0c-fecf7b5ab708"). InnerVolumeSpecName "kube-api-access-xskcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:44:16.006507 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:16.006471    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83fbb6be-6fec-4100-ae0c-fecf7b5ab708-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:44:16.006507 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:16.006501    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xskcz\" (UniqueName: \"kubernetes.io/projected/83fbb6be-6fec-4100-ae0c-fecf7b5ab708-kube-api-access-xskcz\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:44:16.254006 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:16.253923    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5671ec2a-f178-497c-9edd-dd4e11e08c68" path="/var/lib/kubelet/pods/5671ec2a-f178-497c-9edd-dd4e11e08c68/volumes"
Sep 10 16:44:16.369301 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:16.369260    2580 generic.go:356] "Generic (PLEG): container finished" podID="83fbb6be-6fec-4100-ae0c-fecf7b5ab708" containerID="98b328e3ed142cc42dae121eaf268eb6efba09f462bfd4d2d31f5c6809e3efe6" exitCode=0
Sep 10 16:44:16.369505 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:16.369307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x" event={"ID":"83fbb6be-6fec-4100-ae0c-fecf7b5ab708","Type":"ContainerDied","Data":"98b328e3ed142cc42dae121eaf268eb6efba09f462bfd4d2d31f5c6809e3efe6"}
Sep 10 16:44:16.369505 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:16.369350    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"
Sep 10 16:44:16.369505 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:16.369406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x" event={"ID":"83fbb6be-6fec-4100-ae0c-fecf7b5ab708","Type":"ContainerDied","Data":"f9013bad6d0f4b3d1a7ebeae2a63457d317a60ec930f2d7962fb12c7c1055f54"}
Sep 10 16:44:16.369505 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:16.369437    2580 scope.go:117] "RemoveContainer" containerID="98b328e3ed142cc42dae121eaf268eb6efba09f462bfd4d2d31f5c6809e3efe6"
Sep 10 16:44:16.379111 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:16.379091    2580 scope.go:117] "RemoveContainer" containerID="98b328e3ed142cc42dae121eaf268eb6efba09f462bfd4d2d31f5c6809e3efe6"
Sep 10 16:44:16.379422 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:44:16.379399    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b328e3ed142cc42dae121eaf268eb6efba09f462bfd4d2d31f5c6809e3efe6\": container with ID starting with 98b328e3ed142cc42dae121eaf268eb6efba09f462bfd4d2d31f5c6809e3efe6 not found: ID does not exist" containerID="98b328e3ed142cc42dae121eaf268eb6efba09f462bfd4d2d31f5c6809e3efe6"
Sep 10 16:44:16.379504 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:16.379427    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b328e3ed142cc42dae121eaf268eb6efba09f462bfd4d2d31f5c6809e3efe6"} err="failed to get container status \"98b328e3ed142cc42dae121eaf268eb6efba09f462bfd4d2d31f5c6809e3efe6\": rpc error: code = NotFound desc = could not find container \"98b328e3ed142cc42dae121eaf268eb6efba09f462bfd4d2d31f5c6809e3efe6\": container with ID starting with 98b328e3ed142cc42dae121eaf268eb6efba09f462bfd4d2d31f5c6809e3efe6 not found: ID does not exist"
Sep 10 16:44:16.386878 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:16.386850    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"]
Sep 10 16:44:16.390713 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:16.390688    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9d55d-745df856f4-gk72x"]
Sep 10 16:44:18.254493 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:18.254456    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83fbb6be-6fec-4100-ae0c-fecf7b5ab708" path="/var/lib/kubelet/pods/83fbb6be-6fec-4100-ae0c-fecf7b5ab708/volumes"
Sep 10 16:44:26.255344 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.255311    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27"]
Sep 10 16:44:26.255752 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.255640    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5671ec2a-f178-497c-9edd-dd4e11e08c68" containerName="revoke-keys"
Sep 10 16:44:26.255752 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.255652    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5671ec2a-f178-497c-9edd-dd4e11e08c68" containerName="revoke-keys"
Sep 10 16:44:26.255752 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.255685    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83fbb6be-6fec-4100-ae0c-fecf7b5ab708" containerName="payload-pre-processing"
Sep 10 16:44:26.255752 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.255692    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="83fbb6be-6fec-4100-ae0c-fecf7b5ab708" containerName="payload-pre-processing"
Sep 10 16:44:26.255752 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.255747    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5671ec2a-f178-497c-9edd-dd4e11e08c68" containerName="revoke-keys"
Sep 10 16:44:26.255752 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.255755    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="83fbb6be-6fec-4100-ae0c-fecf7b5ab708" containerName="payload-pre-processing"
Sep 10 16:44:26.258434 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.258417    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27"
Sep 10 16:44:26.261064 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.261044    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:44:26.261323 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.261304    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\""
Sep 10 16:44:26.267613 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.267588    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27"]
Sep 10 16:44:26.298948 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.298912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e02c04-662d-47ac-b9dd-687313586c1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27\" (UID: \"d2e02c04-662d-47ac-b9dd-687313586c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27"
Sep 10 16:44:26.399712 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.399675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e02c04-662d-47ac-b9dd-687313586c1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27\" (UID: \"d2e02c04-662d-47ac-b9dd-687313586c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27"
Sep 10 16:44:26.400409 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.400382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e02c04-662d-47ac-b9dd-687313586c1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27\" (UID: \"d2e02c04-662d-47ac-b9dd-687313586c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27"
Sep 10 16:44:26.569846 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.569800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27"
Sep 10 16:44:26.846978 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:26.846887    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27"]
Sep 10 16:44:26.854311 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:44:26.854258    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e02c04_662d_47ac_b9dd_687313586c1a.slice/crio-b2ab9c1c51c558f70d41a6dab504ca6dd72e0ff4191ffd24dfbc4e95f1e8240b WatchSource:0}: Error finding container b2ab9c1c51c558f70d41a6dab504ca6dd72e0ff4191ffd24dfbc4e95f1e8240b: Status 404 returned error can't find the container with id b2ab9c1c51c558f70d41a6dab504ca6dd72e0ff4191ffd24dfbc4e95f1e8240b
Sep 10 16:44:27.428905 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:27.428871    2580 generic.go:356] "Generic (PLEG): container finished" podID="d2e02c04-662d-47ac-b9dd-687313586c1a" containerID="7e2ee2413c58ca5ce607b71322dbb20e7b7e190f4ce0df89aca267fb03dd49f8" exitCode=0
Sep 10 16:44:27.429235 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:27.428919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27" event={"ID":"d2e02c04-662d-47ac-b9dd-687313586c1a","Type":"ContainerDied","Data":"7e2ee2413c58ca5ce607b71322dbb20e7b7e190f4ce0df89aca267fb03dd49f8"}
Sep 10 16:44:27.429235 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:27.428965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27" event={"ID":"d2e02c04-662d-47ac-b9dd-687313586c1a","Type":"ContainerStarted","Data":"b2ab9c1c51c558f70d41a6dab504ca6dd72e0ff4191ffd24dfbc4e95f1e8240b"}
Sep 10 16:44:28.565856 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:28.565824    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27"
Sep 10 16:44:28.619836 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:28.619803    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e02c04-662d-47ac-b9dd-687313586c1a-maas-api-service-ca\") pod \"d2e02c04-662d-47ac-b9dd-687313586c1a\" (UID: \"d2e02c04-662d-47ac-b9dd-687313586c1a\") "
Sep 10 16:44:28.620217 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:28.620193    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e02c04-662d-47ac-b9dd-687313586c1a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2e02c04-662d-47ac-b9dd-687313586c1a" (UID: "d2e02c04-662d-47ac-b9dd-687313586c1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:44:28.721597 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:28.721501    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e02c04-662d-47ac-b9dd-687313586c1a-maas-api-service-ca\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:44:29.439652 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:29.439623    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27"
Sep 10 16:44:29.439652 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:29.439635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27" event={"ID":"d2e02c04-662d-47ac-b9dd-687313586c1a","Type":"ContainerDied","Data":"b2ab9c1c51c558f70d41a6dab504ca6dd72e0ff4191ffd24dfbc4e95f1e8240b"}
Sep 10 16:44:29.439866 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:29.439687    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2ab9c1c51c558f70d41a6dab504ca6dd72e0ff4191ffd24dfbc4e95f1e8240b"
Sep 10 16:44:42.026090 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:42.026054    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27"]
Sep 10 16:44:42.028918 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:42.028882    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95d261-d5h27"]
Sep 10 16:44:42.255985 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:42.255951    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2e02c04-662d-47ac-b9dd-687313586c1a" path="/var/lib/kubelet/pods/d2e02c04-662d-47ac-b9dd-687313586c1a/volumes"
Sep 10 16:44:47.401136 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:44:47.401099    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"]
Sep 10 16:45:00.150565 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.150509    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86d230-29817645-wv4k8"]
Sep 10 16:45:00.150960 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.150889    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e02c04-662d-47ac-b9dd-687313586c1a" containerName="revoke-keys"
Sep 10 16:45:00.150960 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.150903    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e02c04-662d-47ac-b9dd-687313586c1a" containerName="revoke-keys"
Sep 10 16:45:00.150960 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.150959    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e02c04-662d-47ac-b9dd-687313586c1a" containerName="revoke-keys"
Sep 10 16:45:00.153723 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.153706    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817645-c2v9j"]
Sep 10 16:45:00.153839 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.153820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86d230-29817645-wv4k8"
Sep 10 16:45:00.156783 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.156760    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\""
Sep 10 16:45:00.156783 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.156781    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817645-c2v9j"
Sep 10 16:45:00.164324 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.164265    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86d230-29817645-wv4k8"]
Sep 10 16:45:00.165651 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.165633    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86d230-29817645-wv4k8"
Sep 10 16:45:00.166513 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.166493    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817645-c2v9j"]
Sep 10 16:45:00.177976 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.177948    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817645-c2v9j"
Sep 10 16:45:00.447085 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.447060    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86d230-29817645-wv4k8"]
Sep 10 16:45:00.449590 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:45:00.449556    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e7cf4d_90e7_4b53_be04_cc54b776ce08.slice/crio-61fb29227908b8cb4f51a15890e29880d01e89e57dfd2b44122fccc22952d8c4 WatchSource:0}: Error finding container 61fb29227908b8cb4f51a15890e29880d01e89e57dfd2b44122fccc22952d8c4: Status 404 returned error can't find the container with id 61fb29227908b8cb4f51a15890e29880d01e89e57dfd2b44122fccc22952d8c4
Sep 10 16:45:00.463515 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.463483    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817645-c2v9j"]
Sep 10 16:45:00.464702 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:45:00.464675    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3022ce00_cee6_4698_acd3_faf1d42619be.slice/crio-39d5e5c39d1f1b6ee22282acd7036b352f499648732b17e2ba3b37698c5b54cf WatchSource:0}: Error finding container 39d5e5c39d1f1b6ee22282acd7036b352f499648732b17e2ba3b37698c5b54cf: Status 404 returned error can't find the container with id 39d5e5c39d1f1b6ee22282acd7036b352f499648732b17e2ba3b37698c5b54cf
Sep 10 16:45:00.592611 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.592573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817645-c2v9j" event={"ID":"3022ce00-cee6-4698-acd3-faf1d42619be","Type":"ContainerStarted","Data":"39d5e5c39d1f1b6ee22282acd7036b352f499648732b17e2ba3b37698c5b54cf"}
Sep 10 16:45:00.593799 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:00.593776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86d230-29817645-wv4k8" event={"ID":"d2e7cf4d-90e7-4b53-be04-cc54b776ce08","Type":"ContainerStarted","Data":"61fb29227908b8cb4f51a15890e29880d01e89e57dfd2b44122fccc22952d8c4"}
Sep 10 16:45:01.600053 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:01.599955    2580 generic.go:356] "Generic (PLEG): container finished" podID="3022ce00-cee6-4698-acd3-faf1d42619be" containerID="d0ba89e63cf1c9d28b52d427a2591b18f18bee28081a2dc71ae7552e8c8ab97a" exitCode=0
Sep 10 16:45:01.600053 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:01.600043    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817645-c2v9j" event={"ID":"3022ce00-cee6-4698-acd3-faf1d42619be","Type":"ContainerDied","Data":"d0ba89e63cf1c9d28b52d427a2591b18f18bee28081a2dc71ae7552e8c8ab97a"}
Sep 10 16:45:01.601654 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:01.601629    2580 generic.go:356] "Generic (PLEG): container finished" podID="d2e7cf4d-90e7-4b53-be04-cc54b776ce08" containerID="acc805bed9296e4195b0dd1694fdfa8d328adfba948c8d11a4afd5e378f42ddc" exitCode=0
Sep 10 16:45:01.601758 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:01.601670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86d230-29817645-wv4k8" event={"ID":"d2e7cf4d-90e7-4b53-be04-cc54b776ce08","Type":"ContainerDied","Data":"acc805bed9296e4195b0dd1694fdfa8d328adfba948c8d11a4afd5e378f42ddc"}
Sep 10 16:45:02.420784 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.420735    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz" podUID="1e9a46f0-7b2d-4200-a98e-4c9f80c30828" containerName="main" containerID="cri-o://a5b55cd8be8c14658db5744163a02cd3777b0dab8ba63a2a262b6348041f6bc4" gracePeriod=45
Sep 10 16:45:02.609767 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.609732    2580 generic.go:356] "Generic (PLEG): container finished" podID="1e9a46f0-7b2d-4200-a98e-4c9f80c30828" containerID="a5b55cd8be8c14658db5744163a02cd3777b0dab8ba63a2a262b6348041f6bc4" exitCode=0
Sep 10 16:45:02.610250 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.609807    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz" event={"ID":"1e9a46f0-7b2d-4200-a98e-4c9f80c30828","Type":"ContainerDied","Data":"a5b55cd8be8c14658db5744163a02cd3777b0dab8ba63a2a262b6348041f6bc4"}
Sep 10 16:45:02.673901 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.673834    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:45:02.727876 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.727839    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-tmp-dir\") pod \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") "
Sep 10 16:45:02.728073 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.727902    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-model-cache\") pod \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") "
Sep 10 16:45:02.728073 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.727928    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-tls-certs\") pod \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") "
Sep 10 16:45:02.728073 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.727978    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-home\") pod \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") "
Sep 10 16:45:02.728073 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.728015    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd5t6\" (UniqueName: \"kubernetes.io/projected/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-kube-api-access-hd5t6\") pod \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") "
Sep 10 16:45:02.728381 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.728105    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-dshm\") pod \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\" (UID: \"1e9a46f0-7b2d-4200-a98e-4c9f80c30828\") "
Sep 10 16:45:02.728381 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.728102    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e9a46f0-7b2d-4200-a98e-4c9f80c30828" (UID: "1e9a46f0-7b2d-4200-a98e-4c9f80c30828"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:45:02.728503 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.728182    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e9a46f0-7b2d-4200-a98e-4c9f80c30828" (UID: "1e9a46f0-7b2d-4200-a98e-4c9f80c30828"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:45:02.728503 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.728285    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-home" (OuterVolumeSpecName: "home") pod "1e9a46f0-7b2d-4200-a98e-4c9f80c30828" (UID: "1e9a46f0-7b2d-4200-a98e-4c9f80c30828"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:45:02.732880 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.730952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-kube-api-access-hd5t6" (OuterVolumeSpecName: "kube-api-access-hd5t6") pod "1e9a46f0-7b2d-4200-a98e-4c9f80c30828" (UID: "1e9a46f0-7b2d-4200-a98e-4c9f80c30828"). InnerVolumeSpecName "kube-api-access-hd5t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:45:02.732880 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.730974    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-dshm" (OuterVolumeSpecName: "dshm") pod "1e9a46f0-7b2d-4200-a98e-4c9f80c30828" (UID: "1e9a46f0-7b2d-4200-a98e-4c9f80c30828"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:45:02.732880 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.730976    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e9a46f0-7b2d-4200-a98e-4c9f80c30828" (UID: "1e9a46f0-7b2d-4200-a98e-4c9f80c30828"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:45:02.758726 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.758700    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86d230-29817645-wv4k8"
Sep 10 16:45:02.777030 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.776999    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817645-c2v9j"
Sep 10 16:45:02.829359 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.829328    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:02.829359 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.829354    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:02.829359 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.829364    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:02.829633 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.829372    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:02.829633 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.829379    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:02.829633 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:02.829387    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd5t6\" (UniqueName: \"kubernetes.io/projected/1e9a46f0-7b2d-4200-a98e-4c9f80c30828-kube-api-access-hd5t6\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:03.616595 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:03.616556    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"
Sep 10 16:45:03.616595 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:03.616581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz" event={"ID":"1e9a46f0-7b2d-4200-a98e-4c9f80c30828","Type":"ContainerDied","Data":"2a90c0b2ded14198ec5edf39db8966601a3ef471c3186b7577e53e75162708fe"}
Sep 10 16:45:03.617159 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:03.616630    2580 scope.go:117] "RemoveContainer" containerID="a5b55cd8be8c14658db5744163a02cd3777b0dab8ba63a2a262b6348041f6bc4"
Sep 10 16:45:03.618125 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:03.618101    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817645-c2v9j"
Sep 10 16:45:03.618125 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:03.618114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817645-c2v9j" event={"ID":"3022ce00-cee6-4698-acd3-faf1d42619be","Type":"ContainerDied","Data":"39d5e5c39d1f1b6ee22282acd7036b352f499648732b17e2ba3b37698c5b54cf"}
Sep 10 16:45:03.618321 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:03.618169    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39d5e5c39d1f1b6ee22282acd7036b352f499648732b17e2ba3b37698c5b54cf"
Sep 10 16:45:03.619626 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:03.619597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86d230-29817645-wv4k8" event={"ID":"d2e7cf4d-90e7-4b53-be04-cc54b776ce08","Type":"ContainerDied","Data":"61fb29227908b8cb4f51a15890e29880d01e89e57dfd2b44122fccc22952d8c4"}
Sep 10 16:45:03.619740 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:03.619640    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61fb29227908b8cb4f51a15890e29880d01e89e57dfd2b44122fccc22952d8c4"
Sep 10 16:45:03.619740 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:03.619638    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86d230-29817645-wv4k8"
Sep 10 16:45:03.644094 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:03.644063    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"]
Sep 10 16:45:03.648859 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:03.648835    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-95d261/ipp-route-95d261-kserve-6449846b7f-f54sz"]
Sep 10 16:45:04.256697 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:04.256663    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e9a46f0-7b2d-4200-a98e-4c9f80c30828" path="/var/lib/kubelet/pods/1e9a46f0-7b2d-4200-a98e-4c9f80c30828/volumes"
Sep 10 16:45:42.814457 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:42.814420    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86d230-29817645-wv4k8"]
Sep 10 16:45:42.817306 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:42.817285    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86d230-29817645-wv4k8"]
Sep 10 16:45:42.851171 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:42.851138    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"]
Sep 10 16:45:42.851419 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:42.851374    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n" podUID="6894ba38-4323-4453-b1dc-bd90334b6c08" containerName="payload-pre-processing" containerID="cri-o://65eb7542b37f747b28ef223b94c7128203ebec9fef2b8c4f5faad8833b734893" gracePeriod=30
Sep 10 16:45:43.105864 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.105833    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"
Sep 10 16:45:43.295233 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.295201    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6894ba38-4323-4453-b1dc-bd90334b6c08-plugins-config-volume\") pod \"6894ba38-4323-4453-b1dc-bd90334b6c08\" (UID: \"6894ba38-4323-4453-b1dc-bd90334b6c08\") "
Sep 10 16:45:43.295403 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.295259    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkm5m\" (UniqueName: \"kubernetes.io/projected/6894ba38-4323-4453-b1dc-bd90334b6c08-kube-api-access-jkm5m\") pod \"6894ba38-4323-4453-b1dc-bd90334b6c08\" (UID: \"6894ba38-4323-4453-b1dc-bd90334b6c08\") "
Sep 10 16:45:43.295600 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.295578    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6894ba38-4323-4453-b1dc-bd90334b6c08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6894ba38-4323-4453-b1dc-bd90334b6c08" (UID: "6894ba38-4323-4453-b1dc-bd90334b6c08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:45:43.297278 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.297255    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6894ba38-4323-4453-b1dc-bd90334b6c08-kube-api-access-jkm5m" (OuterVolumeSpecName: "kube-api-access-jkm5m") pod "6894ba38-4323-4453-b1dc-bd90334b6c08" (UID: "6894ba38-4323-4453-b1dc-bd90334b6c08"). InnerVolumeSpecName "kube-api-access-jkm5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:45:43.396426 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.396359    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkm5m\" (UniqueName: \"kubernetes.io/projected/6894ba38-4323-4453-b1dc-bd90334b6c08-kube-api-access-jkm5m\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:43.396426 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.396383    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6894ba38-4323-4453-b1dc-bd90334b6c08-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:43.824406 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.824370    2580 generic.go:356] "Generic (PLEG): container finished" podID="6894ba38-4323-4453-b1dc-bd90334b6c08" containerID="65eb7542b37f747b28ef223b94c7128203ebec9fef2b8c4f5faad8833b734893" exitCode=0
Sep 10 16:45:43.824823 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.824418    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"
Sep 10 16:45:43.824823 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.824453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n" event={"ID":"6894ba38-4323-4453-b1dc-bd90334b6c08","Type":"ContainerDied","Data":"65eb7542b37f747b28ef223b94c7128203ebec9fef2b8c4f5faad8833b734893"}
Sep 10 16:45:43.824823 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.824507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n" event={"ID":"6894ba38-4323-4453-b1dc-bd90334b6c08","Type":"ContainerDied","Data":"b8e7d9588778294f1cb7e7a6ad519420377c78617b8b323a145d2121cea2c55e"}
Sep 10 16:45:43.824823 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.824550    2580 scope.go:117] "RemoveContainer" containerID="65eb7542b37f747b28ef223b94c7128203ebec9fef2b8c4f5faad8833b734893"
Sep 10 16:45:43.833293 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.833271    2580 scope.go:117] "RemoveContainer" containerID="65eb7542b37f747b28ef223b94c7128203ebec9fef2b8c4f5faad8833b734893"
Sep 10 16:45:43.833577 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:45:43.833558    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65eb7542b37f747b28ef223b94c7128203ebec9fef2b8c4f5faad8833b734893\": container with ID starting with 65eb7542b37f747b28ef223b94c7128203ebec9fef2b8c4f5faad8833b734893 not found: ID does not exist" containerID="65eb7542b37f747b28ef223b94c7128203ebec9fef2b8c4f5faad8833b734893"
Sep 10 16:45:43.833658 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.833589    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65eb7542b37f747b28ef223b94c7128203ebec9fef2b8c4f5faad8833b734893"} err="failed to get container status \"65eb7542b37f747b28ef223b94c7128203ebec9fef2b8c4f5faad8833b734893\": rpc error: code = NotFound desc = could not find container \"65eb7542b37f747b28ef223b94c7128203ebec9fef2b8c4f5faad8833b734893\": container with ID starting with 65eb7542b37f747b28ef223b94c7128203ebec9fef2b8c4f5faad8833b734893 not found: ID does not exist"
Sep 10 16:45:43.849948 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.849924    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"]
Sep 10 16:45:43.856110 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:43.856093    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86d230-5464c55d6-km55n"]
Sep 10 16:45:44.254066 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:44.253987    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6894ba38-4323-4453-b1dc-bd90334b6c08" path="/var/lib/kubelet/pods/6894ba38-4323-4453-b1dc-bd90334b6c08/volumes"
Sep 10 16:45:44.254291 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:44.254277    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2e7cf4d-90e7-4b53-be04-cc54b776ce08" path="/var/lib/kubelet/pods/d2e7cf4d-90e7-4b53-be04-cc54b776ce08/volumes"
Sep 10 16:45:50.287175 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:50.287103    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"]
Sep 10 16:45:50.287574 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:50.287322    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7" podUID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerName="istio-proxy" containerID="cri-o://fc3b5cb8bdefb6066d899c71ff3fde650fa4af9263f8b2108f88b20d3e748aa2" gracePeriod=30
Sep 10 16:45:55.543823 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.543800    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:45:55.589349 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.589291    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-podinfo\") pod \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") "
Sep 10 16:45:55.589458 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.589349    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txxcg\" (UniqueName: \"kubernetes.io/projected/92367ce3-7f8b-4a38-a92e-ce005b81b74b-kube-api-access-txxcg\") pod \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") "
Sep 10 16:45:55.589458 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.589397    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-credential-socket\") pod \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") "
Sep 10 16:45:55.589458 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.589425    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-workload-certs\") pod \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") "
Sep 10 16:45:55.589458 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.589454    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-token\") pod \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") "
Sep 10 16:45:55.589696 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.589477    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-envoy\") pod \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") "
Sep 10 16:45:55.589768 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.589742    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "92367ce3-7f8b-4a38-a92e-ce005b81b74b" (UID: "92367ce3-7f8b-4a38-a92e-ce005b81b74b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:45:55.589768 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.589758    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istiod-ca-cert\") pod \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") "
Sep 10 16:45:55.589891 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.589769    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "92367ce3-7f8b-4a38-a92e-ce005b81b74b" (UID: "92367ce3-7f8b-4a38-a92e-ce005b81b74b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:45:55.589891 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.589791    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-workload-socket\") pod \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") "
Sep 10 16:45:55.589891 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.589821    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-data\") pod \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\" (UID: \"92367ce3-7f8b-4a38-a92e-ce005b81b74b\") "
Sep 10 16:45:55.590071 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.590049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "92367ce3-7f8b-4a38-a92e-ce005b81b74b" (UID: "92367ce3-7f8b-4a38-a92e-ce005b81b74b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:45:55.590139 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.590101    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-credential-socket\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:55.590139 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.590106    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "92367ce3-7f8b-4a38-a92e-ce005b81b74b" (UID: "92367ce3-7f8b-4a38-a92e-ce005b81b74b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:45:55.590139 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.590122    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-workload-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:55.590357 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.590331    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-data" (OuterVolumeSpecName: "istio-data") pod "92367ce3-7f8b-4a38-a92e-ce005b81b74b" (UID: "92367ce3-7f8b-4a38-a92e-ce005b81b74b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:45:55.591802 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.591774    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-token" (OuterVolumeSpecName: "istio-token") pod "92367ce3-7f8b-4a38-a92e-ce005b81b74b" (UID: "92367ce3-7f8b-4a38-a92e-ce005b81b74b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:45:55.591951 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.591931    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "92367ce3-7f8b-4a38-a92e-ce005b81b74b" (UID: "92367ce3-7f8b-4a38-a92e-ce005b81b74b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:45:55.592021 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.591989    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92367ce3-7f8b-4a38-a92e-ce005b81b74b-kube-api-access-txxcg" (OuterVolumeSpecName: "kube-api-access-txxcg") pod "92367ce3-7f8b-4a38-a92e-ce005b81b74b" (UID: "92367ce3-7f8b-4a38-a92e-ce005b81b74b"). InnerVolumeSpecName "kube-api-access-txxcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:45:55.592021 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.591999    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "92367ce3-7f8b-4a38-a92e-ce005b81b74b" (UID: "92367ce3-7f8b-4a38-a92e-ce005b81b74b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:45:55.690442 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.690420    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-token\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:55.690442 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.690440    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-envoy\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:55.690598 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.690449    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istiod-ca-cert\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:55.690598 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.690456    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-workload-socket\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:55.690598 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.690464    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-data\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:55.690598 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.690472    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92367ce3-7f8b-4a38-a92e-ce005b81b74b-istio-podinfo\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:55.690598 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.690480    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txxcg\" (UniqueName: \"kubernetes.io/projected/92367ce3-7f8b-4a38-a92e-ce005b81b74b-kube-api-access-txxcg\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:45:55.887177 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.887112    2580 generic.go:356] "Generic (PLEG): container finished" podID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerID="fc3b5cb8bdefb6066d899c71ff3fde650fa4af9263f8b2108f88b20d3e748aa2" exitCode=0
Sep 10 16:45:55.887177 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.887165    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"
Sep 10 16:45:55.887326 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.887183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7" event={"ID":"92367ce3-7f8b-4a38-a92e-ce005b81b74b","Type":"ContainerDied","Data":"fc3b5cb8bdefb6066d899c71ff3fde650fa4af9263f8b2108f88b20d3e748aa2"}
Sep 10 16:45:55.887326 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.887224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7" event={"ID":"92367ce3-7f8b-4a38-a92e-ce005b81b74b","Type":"ContainerDied","Data":"508e70e948ced98f8a80c84d74f592eb00df7112d2b9af56c1f1c60c936d56bb"}
Sep 10 16:45:55.887326 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.887245    2580 scope.go:117] "RemoveContainer" containerID="fc3b5cb8bdefb6066d899c71ff3fde650fa4af9263f8b2108f88b20d3e748aa2"
Sep 10 16:45:55.896281 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.896265    2580 scope.go:117] "RemoveContainer" containerID="36836c20829f3cd16cc3409fc2a40a10bb370f5decf968bfc758e97e8d6eea09"
Sep 10 16:45:55.903778 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.903761    2580 scope.go:117] "RemoveContainer" containerID="fc3b5cb8bdefb6066d899c71ff3fde650fa4af9263f8b2108f88b20d3e748aa2"
Sep 10 16:45:55.904028 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:45:55.904013    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3b5cb8bdefb6066d899c71ff3fde650fa4af9263f8b2108f88b20d3e748aa2\": container with ID starting with fc3b5cb8bdefb6066d899c71ff3fde650fa4af9263f8b2108f88b20d3e748aa2 not found: ID does not exist" containerID="fc3b5cb8bdefb6066d899c71ff3fde650fa4af9263f8b2108f88b20d3e748aa2"
Sep 10 16:45:55.904088 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.904033    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3b5cb8bdefb6066d899c71ff3fde650fa4af9263f8b2108f88b20d3e748aa2"} err="failed to get container status \"fc3b5cb8bdefb6066d899c71ff3fde650fa4af9263f8b2108f88b20d3e748aa2\": rpc error: code = NotFound desc = could not find container \"fc3b5cb8bdefb6066d899c71ff3fde650fa4af9263f8b2108f88b20d3e748aa2\": container with ID starting with fc3b5cb8bdefb6066d899c71ff3fde650fa4af9263f8b2108f88b20d3e748aa2 not found: ID does not exist"
Sep 10 16:45:55.904088 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.904056    2580 scope.go:117] "RemoveContainer" containerID="36836c20829f3cd16cc3409fc2a40a10bb370f5decf968bfc758e97e8d6eea09"
Sep 10 16:45:55.904316 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:45:55.904290    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36836c20829f3cd16cc3409fc2a40a10bb370f5decf968bfc758e97e8d6eea09\": container with ID starting with 36836c20829f3cd16cc3409fc2a40a10bb370f5decf968bfc758e97e8d6eea09 not found: ID does not exist" containerID="36836c20829f3cd16cc3409fc2a40a10bb370f5decf968bfc758e97e8d6eea09"
Sep 10 16:45:55.904407 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.904322    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36836c20829f3cd16cc3409fc2a40a10bb370f5decf968bfc758e97e8d6eea09"} err="failed to get container status \"36836c20829f3cd16cc3409fc2a40a10bb370f5decf968bfc758e97e8d6eea09\": rpc error: code = NotFound desc = could not find container \"36836c20829f3cd16cc3409fc2a40a10bb370f5decf968bfc758e97e8d6eea09\": container with ID starting with 36836c20829f3cd16cc3409fc2a40a10bb370f5decf968bfc758e97e8d6eea09 not found: ID does not exist"
Sep 10 16:45:55.909648 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.909629    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"]
Sep 10 16:45:55.915456 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:55.915435    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-86d230-openshift-default-868bf56bb6-7k9x7"]
Sep 10 16:45:56.254485 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:45:56.254404    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" path="/var/lib/kubelet/pods/92367ce3-7f8b-4a38-a92e-ce005b81b74b/volumes"
Sep 10 16:46:01.302365 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.302321    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"]
Sep 10 16:46:01.302767 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.302640    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks" podUID="739e13de-b90f-434d-805d-e75b1e6c9b2e" containerName="payload-pre-processing" containerID="cri-o://42b4e8d4c88ce5c79017c437f5aff1620bdd86e621cb19d8d2d8a9f97185e202" gracePeriod=30
Sep 10 16:46:01.553851 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.553797    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"
Sep 10 16:46:01.638289 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.638262    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739e13de-b90f-434d-805d-e75b1e6c9b2e-plugins-config-volume\") pod \"739e13de-b90f-434d-805d-e75b1e6c9b2e\" (UID: \"739e13de-b90f-434d-805d-e75b1e6c9b2e\") "
Sep 10 16:46:01.638438 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.638331    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c4wf\" (UniqueName: \"kubernetes.io/projected/739e13de-b90f-434d-805d-e75b1e6c9b2e-kube-api-access-2c4wf\") pod \"739e13de-b90f-434d-805d-e75b1e6c9b2e\" (UID: \"739e13de-b90f-434d-805d-e75b1e6c9b2e\") "
Sep 10 16:46:01.638676 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.638655    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739e13de-b90f-434d-805d-e75b1e6c9b2e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "739e13de-b90f-434d-805d-e75b1e6c9b2e" (UID: "739e13de-b90f-434d-805d-e75b1e6c9b2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:46:01.640405 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.640378    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739e13de-b90f-434d-805d-e75b1e6c9b2e-kube-api-access-2c4wf" (OuterVolumeSpecName: "kube-api-access-2c4wf") pod "739e13de-b90f-434d-805d-e75b1e6c9b2e" (UID: "739e13de-b90f-434d-805d-e75b1e6c9b2e"). InnerVolumeSpecName "kube-api-access-2c4wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:46:01.739769 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.739743    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739e13de-b90f-434d-805d-e75b1e6c9b2e-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:46:01.739769 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.739764    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c4wf\" (UniqueName: \"kubernetes.io/projected/739e13de-b90f-434d-805d-e75b1e6c9b2e-kube-api-access-2c4wf\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:46:01.915940 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.915873    2580 generic.go:356] "Generic (PLEG): container finished" podID="739e13de-b90f-434d-805d-e75b1e6c9b2e" containerID="42b4e8d4c88ce5c79017c437f5aff1620bdd86e621cb19d8d2d8a9f97185e202" exitCode=0
Sep 10 16:46:01.915940 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.915914    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"
Sep 10 16:46:01.915940 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.915934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks" event={"ID":"739e13de-b90f-434d-805d-e75b1e6c9b2e","Type":"ContainerDied","Data":"42b4e8d4c88ce5c79017c437f5aff1620bdd86e621cb19d8d2d8a9f97185e202"}
Sep 10 16:46:01.916115 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.915973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks" event={"ID":"739e13de-b90f-434d-805d-e75b1e6c9b2e","Type":"ContainerDied","Data":"238b110ab520285fbff225011e7a108bd4f56723ee1fb94be3010090dbd5ceaa"}
Sep 10 16:46:01.916115 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.915996    2580 scope.go:117] "RemoveContainer" containerID="42b4e8d4c88ce5c79017c437f5aff1620bdd86e621cb19d8d2d8a9f97185e202"
Sep 10 16:46:01.924511 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.924492    2580 scope.go:117] "RemoveContainer" containerID="42b4e8d4c88ce5c79017c437f5aff1620bdd86e621cb19d8d2d8a9f97185e202"
Sep 10 16:46:01.924767 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:46:01.924750    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b4e8d4c88ce5c79017c437f5aff1620bdd86e621cb19d8d2d8a9f97185e202\": container with ID starting with 42b4e8d4c88ce5c79017c437f5aff1620bdd86e621cb19d8d2d8a9f97185e202 not found: ID does not exist" containerID="42b4e8d4c88ce5c79017c437f5aff1620bdd86e621cb19d8d2d8a9f97185e202"
Sep 10 16:46:01.924821 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.924775    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b4e8d4c88ce5c79017c437f5aff1620bdd86e621cb19d8d2d8a9f97185e202"} err="failed to get container status \"42b4e8d4c88ce5c79017c437f5aff1620bdd86e621cb19d8d2d8a9f97185e202\": rpc error: code = NotFound desc = could not find container \"42b4e8d4c88ce5c79017c437f5aff1620bdd86e621cb19d8d2d8a9f97185e202\": container with ID starting with 42b4e8d4c88ce5c79017c437f5aff1620bdd86e621cb19d8d2d8a9f97185e202 not found: ID does not exist"
Sep 10 16:46:01.937240 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.937214    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"]
Sep 10 16:46:01.949706 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:01.949683    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1925d8-6f74dfc778-qvkks"]
Sep 10 16:46:02.254104 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:02.254025    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="739e13de-b90f-434d-805d-e75b1e6c9b2e" path="/var/lib/kubelet/pods/739e13de-b90f-434d-805d-e75b1e6c9b2e/volumes"
Sep 10 16:46:19.752735 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:19.752673    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"]
Sep 10 16:46:19.753191 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:19.752951    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh" podUID="2832d858-e5bd-45a5-9f6c-3b509a8a454c" containerName="payload-processing" containerID="cri-o://a0e4083467ebc4f389fbb4fa31cd86f4117763cadcb9336a8aa3fb3adcd39c4a" gracePeriod=30
Sep 10 16:46:19.760912 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:19.760875    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"]
Sep 10 16:46:19.761125 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:19.761101    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj" podUID="a696f483-bbba-4458-a279-9b0aa2cf5733" containerName="payload-pre-processing" containerID="cri-o://62cbec8170d66767ea66beed45c2d7572339f2e69a431f4be8abd0754c8d2172" gracePeriod=30
Sep 10 16:46:19.997306 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:19.997278    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"
Sep 10 16:46:20.004485 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.004395    2580 generic.go:356] "Generic (PLEG): container finished" podID="2832d858-e5bd-45a5-9f6c-3b509a8a454c" containerID="a0e4083467ebc4f389fbb4fa31cd86f4117763cadcb9336a8aa3fb3adcd39c4a" exitCode=0
Sep 10 16:46:20.004485 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.004438    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh" event={"ID":"2832d858-e5bd-45a5-9f6c-3b509a8a454c","Type":"ContainerDied","Data":"a0e4083467ebc4f389fbb4fa31cd86f4117763cadcb9336a8aa3fb3adcd39c4a"}
Sep 10 16:46:20.004485 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.004462    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"
Sep 10 16:46:20.004851 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.004491    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh" event={"ID":"2832d858-e5bd-45a5-9f6c-3b509a8a454c","Type":"ContainerDied","Data":"483bb4ca3e776cbd73166f73fbc796b71f2217ffe49ade6e8b6d37b008af47ec"}
Sep 10 16:46:20.004851 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.004524    2580 scope.go:117] "RemoveContainer" containerID="a0e4083467ebc4f389fbb4fa31cd86f4117763cadcb9336a8aa3fb3adcd39c4a"
Sep 10 16:46:20.006398 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.006376    2580 generic.go:356] "Generic (PLEG): container finished" podID="a696f483-bbba-4458-a279-9b0aa2cf5733" containerID="62cbec8170d66767ea66beed45c2d7572339f2e69a431f4be8abd0754c8d2172" exitCode=0
Sep 10 16:46:20.006518 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.006450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj" event={"ID":"a696f483-bbba-4458-a279-9b0aa2cf5733","Type":"ContainerDied","Data":"62cbec8170d66767ea66beed45c2d7572339f2e69a431f4be8abd0754c8d2172"}
Sep 10 16:46:20.013500 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.013461    2580 scope.go:117] "RemoveContainer" containerID="a0e4083467ebc4f389fbb4fa31cd86f4117763cadcb9336a8aa3fb3adcd39c4a"
Sep 10 16:46:20.013838 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:46:20.013813    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e4083467ebc4f389fbb4fa31cd86f4117763cadcb9336a8aa3fb3adcd39c4a\": container with ID starting with a0e4083467ebc4f389fbb4fa31cd86f4117763cadcb9336a8aa3fb3adcd39c4a not found: ID does not exist" containerID="a0e4083467ebc4f389fbb4fa31cd86f4117763cadcb9336a8aa3fb3adcd39c4a"
Sep 10 16:46:20.013926 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.013850    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e4083467ebc4f389fbb4fa31cd86f4117763cadcb9336a8aa3fb3adcd39c4a"} err="failed to get container status \"a0e4083467ebc4f389fbb4fa31cd86f4117763cadcb9336a8aa3fb3adcd39c4a\": rpc error: code = NotFound desc = could not find container \"a0e4083467ebc4f389fbb4fa31cd86f4117763cadcb9336a8aa3fb3adcd39c4a\": container with ID starting with a0e4083467ebc4f389fbb4fa31cd86f4117763cadcb9336a8aa3fb3adcd39c4a not found: ID does not exist"
Sep 10 16:46:20.031631 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.031612    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"
Sep 10 16:46:20.090665 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.090637    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a696f483-bbba-4458-a279-9b0aa2cf5733-plugins-config-volume\") pod \"a696f483-bbba-4458-a279-9b0aa2cf5733\" (UID: \"a696f483-bbba-4458-a279-9b0aa2cf5733\") "
Sep 10 16:46:20.090846 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.090718    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf56v\" (UniqueName: \"kubernetes.io/projected/a696f483-bbba-4458-a279-9b0aa2cf5733-kube-api-access-sf56v\") pod \"a696f483-bbba-4458-a279-9b0aa2cf5733\" (UID: \"a696f483-bbba-4458-a279-9b0aa2cf5733\") "
Sep 10 16:46:20.090846 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.090758    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2832d858-e5bd-45a5-9f6c-3b509a8a454c-plugins-config-volume\") pod \"2832d858-e5bd-45a5-9f6c-3b509a8a454c\" (UID: \"2832d858-e5bd-45a5-9f6c-3b509a8a454c\") "
Sep 10 16:46:20.090846 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.090789    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq5sc\" (UniqueName: \"kubernetes.io/projected/2832d858-e5bd-45a5-9f6c-3b509a8a454c-kube-api-access-dq5sc\") pod \"2832d858-e5bd-45a5-9f6c-3b509a8a454c\" (UID: \"2832d858-e5bd-45a5-9f6c-3b509a8a454c\") "
Sep 10 16:46:20.091109 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.091083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a696f483-bbba-4458-a279-9b0aa2cf5733-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a696f483-bbba-4458-a279-9b0aa2cf5733" (UID: "a696f483-bbba-4458-a279-9b0aa2cf5733"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:46:20.091234 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.091142    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2832d858-e5bd-45a5-9f6c-3b509a8a454c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2832d858-e5bd-45a5-9f6c-3b509a8a454c" (UID: "2832d858-e5bd-45a5-9f6c-3b509a8a454c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:46:20.092946 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.092918    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2832d858-e5bd-45a5-9f6c-3b509a8a454c-kube-api-access-dq5sc" (OuterVolumeSpecName: "kube-api-access-dq5sc") pod "2832d858-e5bd-45a5-9f6c-3b509a8a454c" (UID: "2832d858-e5bd-45a5-9f6c-3b509a8a454c"). InnerVolumeSpecName "kube-api-access-dq5sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:46:20.093026 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.092973    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a696f483-bbba-4458-a279-9b0aa2cf5733-kube-api-access-sf56v" (OuterVolumeSpecName: "kube-api-access-sf56v") pod "a696f483-bbba-4458-a279-9b0aa2cf5733" (UID: "a696f483-bbba-4458-a279-9b0aa2cf5733"). InnerVolumeSpecName "kube-api-access-sf56v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:46:20.191980 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.191940    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2832d858-e5bd-45a5-9f6c-3b509a8a454c-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:46:20.191980 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.191974    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq5sc\" (UniqueName: \"kubernetes.io/projected/2832d858-e5bd-45a5-9f6c-3b509a8a454c-kube-api-access-dq5sc\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:46:20.191980 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.191983    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a696f483-bbba-4458-a279-9b0aa2cf5733-plugins-config-volume\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:46:20.191980 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.191993    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf56v\" (UniqueName: \"kubernetes.io/projected/a696f483-bbba-4458-a279-9b0aa2cf5733-kube-api-access-sf56v\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:46:20.328577 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.328525    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"]
Sep 10 16:46:20.341799 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:20.341771    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3ccbd4-7ff5f44647-jtqxh"]
Sep 10 16:46:21.014330 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:21.014298    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"
Sep 10 16:46:21.014805 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:21.014340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj" event={"ID":"a696f483-bbba-4458-a279-9b0aa2cf5733","Type":"ContainerDied","Data":"ae441a79a966f56795bcefed6f41646f8fcf743b507084d6416dfa9ab1e98e9e"}
Sep 10 16:46:21.014805 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:21.014403    2580 scope.go:117] "RemoveContainer" containerID="62cbec8170d66767ea66beed45c2d7572339f2e69a431f4be8abd0754c8d2172"
Sep 10 16:46:21.039480 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:21.039454    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"]
Sep 10 16:46:21.043191 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:21.043168    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3ccbd4-58f4d58b54-vggsj"]
Sep 10 16:46:22.255668 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:22.255628    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2832d858-e5bd-45a5-9f6c-3b509a8a454c" path="/var/lib/kubelet/pods/2832d858-e5bd-45a5-9f6c-3b509a8a454c/volumes"
Sep 10 16:46:22.256032 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:46:22.255997    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a696f483-bbba-4458-a279-9b0aa2cf5733" path="/var/lib/kubelet/pods/a696f483-bbba-4458-a279-9b0aa2cf5733/volumes"
Sep 10 16:47:22.539667 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:47:22.539627    2580 scope.go:117] "RemoveContainer" containerID="3697714744df2340c275e8af989dfc0b0d62aae211e1c004c9f0e681d10ec409"
Sep 10 16:47:22.547880 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:47:22.547859    2580 scope.go:117] "RemoveContainer" containerID="eeffdeb6ec9dca8370f269f704560312dc3e4e51126f16f36f6e43045229b317"
Sep 10 16:48:22.569154 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:48:22.569119    2580 scope.go:117] "RemoveContainer" containerID="8a01976a85fbe20ba4c84e7dae8551caaa8fc6b40c46b24753386039954a2622"
Sep 10 16:48:22.577222 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:48:22.577201    2580 scope.go:117] "RemoveContainer" containerID="c2c1c34a4de3cd70c4f6a13b6a4c777bd5e11bbc421aaa0c13bd90d644989999"
Sep 10 16:50:22.605566 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:22.605460    2580 scope.go:117] "RemoveContainer" containerID="9d7bf12eb02c269c842e80b502b87f43dcb342df1ba503c5b12a2e076cade601"
Sep 10 16:50:42.930460 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.930426    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"]
Sep 10 16:50:42.930935 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.930860    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3022ce00-cee6-4698-acd3-faf1d42619be" containerName="cleanup"
Sep 10 16:50:42.930935 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.930877    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3022ce00-cee6-4698-acd3-faf1d42619be" containerName="cleanup"
Sep 10 16:50:42.930935 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.930897    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a696f483-bbba-4458-a279-9b0aa2cf5733" containerName="payload-pre-processing"
Sep 10 16:50:42.930935 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.930906    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a696f483-bbba-4458-a279-9b0aa2cf5733" containerName="payload-pre-processing"
Sep 10 16:50:42.930935 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.930916    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerName="istio-proxy"
Sep 10 16:50:42.930935 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.930924    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerName="istio-proxy"
Sep 10 16:50:42.930935 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.930932    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerName="istio-proxy"
Sep 10 16:50:42.931162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.930942    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerName="istio-proxy"
Sep 10 16:50:42.931162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.930960    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2832d858-e5bd-45a5-9f6c-3b509a8a454c" containerName="payload-processing"
Sep 10 16:50:42.931162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.930969    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2832d858-e5bd-45a5-9f6c-3b509a8a454c" containerName="payload-processing"
Sep 10 16:50:42.931162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.930987    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6894ba38-4323-4453-b1dc-bd90334b6c08" containerName="payload-pre-processing"
Sep 10 16:50:42.931162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.930996    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6894ba38-4323-4453-b1dc-bd90334b6c08" containerName="payload-pre-processing"
Sep 10 16:50:42.931162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.931005    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e9a46f0-7b2d-4200-a98e-4c9f80c30828" containerName="main"
Sep 10 16:50:42.931162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.931014    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e9a46f0-7b2d-4200-a98e-4c9f80c30828" containerName="main"
Sep 10 16:50:42.931162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.931028    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="739e13de-b90f-434d-805d-e75b1e6c9b2e" containerName="payload-pre-processing"
Sep 10 16:50:42.931162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.931038    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="739e13de-b90f-434d-805d-e75b1e6c9b2e" containerName="payload-pre-processing"
Sep 10 16:50:42.931162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.931050    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e7cf4d-90e7-4b53-be04-cc54b776ce08" containerName="cleanup"
Sep 10 16:50:42.931162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.931059    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e7cf4d-90e7-4b53-be04-cc54b776ce08" containerName="cleanup"
Sep 10 16:50:42.931162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.931153    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="739e13de-b90f-434d-805d-e75b1e6c9b2e" containerName="payload-pre-processing"
Sep 10 16:50:42.931162 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.931165    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerName="istio-proxy"
Sep 10 16:50:42.931561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.931174    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6894ba38-4323-4453-b1dc-bd90334b6c08" containerName="payload-pre-processing"
Sep 10 16:50:42.931561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.931185    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e7cf4d-90e7-4b53-be04-cc54b776ce08" containerName="cleanup"
Sep 10 16:50:42.931561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.931192    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2832d858-e5bd-45a5-9f6c-3b509a8a454c" containerName="payload-processing"
Sep 10 16:50:42.931561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.931199    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3022ce00-cee6-4698-acd3-faf1d42619be" containerName="cleanup"
Sep 10 16:50:42.931561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.931207    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e9a46f0-7b2d-4200-a98e-4c9f80c30828" containerName="main"
Sep 10 16:50:42.931561 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.931214    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a696f483-bbba-4458-a279-9b0aa2cf5733" containerName="payload-pre-processing"
Sep 10 16:50:42.934372 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.934352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:42.936467 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.936446    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3d45f766-kserve-self-signed-certs\""
Sep 10 16:50:42.943465 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:42.943439    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"]
Sep 10 16:50:43.029196 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.029167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e5302c-3f82-4988-9995-56ab70a0f24c-tls-certs\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.029303 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.029215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-model-cache\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.029303 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.029292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-dshm\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.029400 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.029320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-tmp-dir\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.029400 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.029344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-home\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.029400 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.029369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx4ck\" (UniqueName: \"kubernetes.io/projected/95e5302c-3f82-4988-9995-56ab70a0f24c-kube-api-access-sx4ck\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.130439 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.130403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e5302c-3f82-4988-9995-56ab70a0f24c-tls-certs\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.130629 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.130460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-model-cache\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.130629 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.130510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-dshm\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.130629 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.130559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-tmp-dir\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.130629 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.130595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-home\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.130855 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.130667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx4ck\" (UniqueName: \"kubernetes.io/projected/95e5302c-3f82-4988-9995-56ab70a0f24c-kube-api-access-sx4ck\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.130961 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.130942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-home\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.131014 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.130983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-model-cache\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.131051 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.131005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-tmp-dir\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.132806 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.132785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-dshm\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.132905 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.132877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e5302c-3f82-4988-9995-56ab70a0f24c-tls-certs\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.138570 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.138550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx4ck\" (UniqueName: \"kubernetes.io/projected/95e5302c-3f82-4988-9995-56ab70a0f24c-kube-api-access-sx4ck\") pod \"e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") " pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.244116 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.244045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:43.514314 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:43.514287    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"]
Sep 10 16:50:43.516049 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:50:43.516024    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95e5302c_3f82_4988_9995_56ab70a0f24c.slice/crio-fd141748a5c3f0ba4c37ebfc50b2b1521cda01bfd35cb3cca2d5e4ad21493ae4 WatchSource:0}: Error finding container fd141748a5c3f0ba4c37ebfc50b2b1521cda01bfd35cb3cca2d5e4ad21493ae4: Status 404 returned error can't find the container with id fd141748a5c3f0ba4c37ebfc50b2b1521cda01bfd35cb3cca2d5e4ad21493ae4
Sep 10 16:50:44.172357 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:44.172323    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl" event={"ID":"95e5302c-3f82-4988-9995-56ab70a0f24c","Type":"ContainerStarted","Data":"d0d2d9b8349d9d2679f852db65d2b0624e544e92e7277d47f1987396d3982531"}
Sep 10 16:50:44.172357 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:44.172363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl" event={"ID":"95e5302c-3f82-4988-9995-56ab70a0f24c","Type":"ContainerStarted","Data":"fd141748a5c3f0ba4c37ebfc50b2b1521cda01bfd35cb3cca2d5e4ad21493ae4"}
Sep 10 16:50:44.190249 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:44.190207    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl" podStartSLOduration=2.190191164 podStartE2EDuration="2.190191164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:50:44.18905608 +0000 UTC m=+2002.390098482" watchObservedRunningTime="2026-09-10 16:50:44.190191164 +0000 UTC m=+2002.391233574"
Sep 10 16:50:53.244297 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:53.244252    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:53.244752 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:53.244448    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:53.257040 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:53.257019    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:54.226251 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:54.226216    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:50:57.038571 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.038514    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"]
Sep 10 16:50:57.039029 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.038869    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="92367ce3-7f8b-4a38-a92e-ce005b81b74b" containerName="istio-proxy"
Sep 10 16:50:57.041761 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.041737    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.043977 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.043959    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-3d45f766-kserve-self-signed-certs\""
Sep 10 16:50:57.050714 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.050692    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"]
Sep 10 16:50:57.138750 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.138727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfqgq\" (UniqueName: \"kubernetes.io/projected/1d378e21-60e7-469f-9582-e58cd2e8086e-kube-api-access-sfqgq\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.138868 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.138760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-tmp-dir\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.138868 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.138802    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-dshm\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.138868 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.138825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d378e21-60e7-469f-9582-e58cd2e8086e-tls-certs\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.138868 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.138857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-home\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.139046 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.138876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-model-cache\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.239361 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.239336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-dshm\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.239497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.239369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d378e21-60e7-469f-9582-e58cd2e8086e-tls-certs\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.239497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.239414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-home\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.239497 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.239440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-model-cache\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.239690 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.239503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfqgq\" (UniqueName: \"kubernetes.io/projected/1d378e21-60e7-469f-9582-e58cd2e8086e-kube-api-access-sfqgq\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.239690 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.239576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-tmp-dir\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.239869 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.239851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-home\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.239930 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.239885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-model-cache\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.239930 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.239923    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-tmp-dir\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.241723 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.241703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-dshm\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.241880 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.241865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d378e21-60e7-469f-9582-e58cd2e8086e-tls-certs\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.247524 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.247499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfqgq\" (UniqueName: \"kubernetes.io/projected/1d378e21-60e7-469f-9582-e58cd2e8086e-kube-api-access-sfqgq\") pod \"e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") " pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.352898 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.352799    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:50:57.623381 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:57.623313    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"]
Sep 10 16:50:57.626687 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:50:57.626655    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d378e21_60e7_469f_9582_e58cd2e8086e.slice/crio-f75457ad2ce5d0ef75e4e802899dc77a2fdb7a70e6789d72b084197f5a59805b WatchSource:0}: Error finding container f75457ad2ce5d0ef75e4e802899dc77a2fdb7a70e6789d72b084197f5a59805b: Status 404 returned error can't find the container with id f75457ad2ce5d0ef75e4e802899dc77a2fdb7a70e6789d72b084197f5a59805b
Sep 10 16:50:58.233089 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:58.233053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq" event={"ID":"1d378e21-60e7-469f-9582-e58cd2e8086e","Type":"ContainerStarted","Data":"66bf39727b6ddc6912eb68b89bba11a3257b56425bd0469e3e11722d575e74a1"}
Sep 10 16:50:58.233446 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:58.233097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq" event={"ID":"1d378e21-60e7-469f-9582-e58cd2e8086e","Type":"ContainerStarted","Data":"f75457ad2ce5d0ef75e4e802899dc77a2fdb7a70e6789d72b084197f5a59805b"}
Sep 10 16:50:58.251347 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:50:58.251287    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq" podStartSLOduration=1.251269467 podStartE2EDuration="1.251269467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:50:58.249970718 +0000 UTC m=+2016.451013129" watchObservedRunningTime="2026-09-10 16:50:58.251269467 +0000 UTC m=+2016.452311881"
Sep 10 16:51:07.353147 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:07.353116    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:51:07.353562 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:07.353159    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:51:07.365456 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:07.365434    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:51:08.285479 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:08.285446    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:51:22.622372 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:22.622340    2580 scope.go:117] "RemoveContainer" containerID="acc805bed9296e4195b0dd1694fdfa8d328adfba948c8d11a4afd5e378f42ddc"
Sep 10 16:51:22.629825 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:22.629806    2580 scope.go:117] "RemoveContainer" containerID="7e2ee2413c58ca5ce607b71322dbb20e7b7e190f4ce0df89aca267fb03dd49f8"
Sep 10 16:51:44.358861 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:44.358830    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"]
Sep 10 16:51:45.378286 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:45.378253    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"]
Sep 10 16:51:59.377964 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.377896    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl" podUID="95e5302c-3f82-4988-9995-56ab70a0f24c" containerName="main" containerID="cri-o://d0d2d9b8349d9d2679f852db65d2b0624e544e92e7277d47f1987396d3982531" gracePeriod=45
Sep 10 16:51:59.623557 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.623519    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:51:59.634763 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.634712    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx4ck\" (UniqueName: \"kubernetes.io/projected/95e5302c-3f82-4988-9995-56ab70a0f24c-kube-api-access-sx4ck\") pod \"95e5302c-3f82-4988-9995-56ab70a0f24c\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") "
Sep 10 16:51:59.634867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.634762    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-home\") pod \"95e5302c-3f82-4988-9995-56ab70a0f24c\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") "
Sep 10 16:51:59.634867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.634792    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-dshm\") pod \"95e5302c-3f82-4988-9995-56ab70a0f24c\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") "
Sep 10 16:51:59.634867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.634810    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-tmp-dir\") pod \"95e5302c-3f82-4988-9995-56ab70a0f24c\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") "
Sep 10 16:51:59.634867 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.634838    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-model-cache\") pod \"95e5302c-3f82-4988-9995-56ab70a0f24c\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") "
Sep 10 16:51:59.635054 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.634880    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e5302c-3f82-4988-9995-56ab70a0f24c-tls-certs\") pod \"95e5302c-3f82-4988-9995-56ab70a0f24c\" (UID: \"95e5302c-3f82-4988-9995-56ab70a0f24c\") "
Sep 10 16:51:59.635125 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.635072    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95e5302c-3f82-4988-9995-56ab70a0f24c" (UID: "95e5302c-3f82-4988-9995-56ab70a0f24c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:51:59.635184 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.635131    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-home" (OuterVolumeSpecName: "home") pod "95e5302c-3f82-4988-9995-56ab70a0f24c" (UID: "95e5302c-3f82-4988-9995-56ab70a0f24c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:51:59.635184 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.635153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-model-cache" (OuterVolumeSpecName: "model-cache") pod "95e5302c-3f82-4988-9995-56ab70a0f24c" (UID: "95e5302c-3f82-4988-9995-56ab70a0f24c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:51:59.636759 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.636737    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-dshm" (OuterVolumeSpecName: "dshm") pod "95e5302c-3f82-4988-9995-56ab70a0f24c" (UID: "95e5302c-3f82-4988-9995-56ab70a0f24c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:51:59.636860 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.636796    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e5302c-3f82-4988-9995-56ab70a0f24c-kube-api-access-sx4ck" (OuterVolumeSpecName: "kube-api-access-sx4ck") pod "95e5302c-3f82-4988-9995-56ab70a0f24c" (UID: "95e5302c-3f82-4988-9995-56ab70a0f24c"). InnerVolumeSpecName "kube-api-access-sx4ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:51:59.636909 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.636897    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e5302c-3f82-4988-9995-56ab70a0f24c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95e5302c-3f82-4988-9995-56ab70a0f24c" (UID: "95e5302c-3f82-4988-9995-56ab70a0f24c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:51:59.735449 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.735419    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e5302c-3f82-4988-9995-56ab70a0f24c-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:51:59.735449 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.735446    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx4ck\" (UniqueName: \"kubernetes.io/projected/95e5302c-3f82-4988-9995-56ab70a0f24c-kube-api-access-sx4ck\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:51:59.735634 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.735456    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:51:59.735634 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.735464    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:51:59.735634 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.735471    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:51:59.735634 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:51:59.735478    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e5302c-3f82-4988-9995-56ab70a0f24c-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:52:00.398140 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.398103    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq" podUID="1d378e21-60e7-469f-9582-e58cd2e8086e" containerName="main" containerID="cri-o://66bf39727b6ddc6912eb68b89bba11a3257b56425bd0469e3e11722d575e74a1" gracePeriod=45
Sep 10 16:52:00.490204 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.490174    2580 generic.go:356] "Generic (PLEG): container finished" podID="95e5302c-3f82-4988-9995-56ab70a0f24c" containerID="d0d2d9b8349d9d2679f852db65d2b0624e544e92e7277d47f1987396d3982531" exitCode=0
Sep 10 16:52:00.490349 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.490217    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"
Sep 10 16:52:00.490349 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.490263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl" event={"ID":"95e5302c-3f82-4988-9995-56ab70a0f24c","Type":"ContainerDied","Data":"d0d2d9b8349d9d2679f852db65d2b0624e544e92e7277d47f1987396d3982531"}
Sep 10 16:52:00.490349 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.490304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl" event={"ID":"95e5302c-3f82-4988-9995-56ab70a0f24c","Type":"ContainerDied","Data":"fd141748a5c3f0ba4c37ebfc50b2b1521cda01bfd35cb3cca2d5e4ad21493ae4"}
Sep 10 16:52:00.490349 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.490323    2580 scope.go:117] "RemoveContainer" containerID="d0d2d9b8349d9d2679f852db65d2b0624e544e92e7277d47f1987396d3982531"
Sep 10 16:52:00.506632 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.506602    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"]
Sep 10 16:52:00.509667 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.509646    2580 scope.go:117] "RemoveContainer" containerID="d0d2d9b8349d9d2679f852db65d2b0624e544e92e7277d47f1987396d3982531"
Sep 10 16:52:00.509994 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:52:00.509972    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d2d9b8349d9d2679f852db65d2b0624e544e92e7277d47f1987396d3982531\": container with ID starting with d0d2d9b8349d9d2679f852db65d2b0624e544e92e7277d47f1987396d3982531 not found: ID does not exist" containerID="d0d2d9b8349d9d2679f852db65d2b0624e544e92e7277d47f1987396d3982531"
Sep 10 16:52:00.510089 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.510002    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d2d9b8349d9d2679f852db65d2b0624e544e92e7277d47f1987396d3982531"} err="failed to get container status \"d0d2d9b8349d9d2679f852db65d2b0624e544e92e7277d47f1987396d3982531\": rpc error: code = NotFound desc = could not find container \"d0d2d9b8349d9d2679f852db65d2b0624e544e92e7277d47f1987396d3982531\": container with ID starting with d0d2d9b8349d9d2679f852db65d2b0624e544e92e7277d47f1987396d3982531 not found: ID does not exist"
Sep 10 16:52:00.510742 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.510723    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3d45f766-kserve-6fbbff57f9-mxgbl"]
Sep 10 16:52:00.643525 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.643497    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:52:00.743303 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.743274    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-model-cache\") pod \"1d378e21-60e7-469f-9582-e58cd2e8086e\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") "
Sep 10 16:52:00.743463 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.743322    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-home\") pod \"1d378e21-60e7-469f-9582-e58cd2e8086e\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") "
Sep 10 16:52:00.743463 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.743353    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-dshm\") pod \"1d378e21-60e7-469f-9582-e58cd2e8086e\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") "
Sep 10 16:52:00.743463 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.743382    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d378e21-60e7-469f-9582-e58cd2e8086e-tls-certs\") pod \"1d378e21-60e7-469f-9582-e58cd2e8086e\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") "
Sep 10 16:52:00.743463 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.743415    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfqgq\" (UniqueName: \"kubernetes.io/projected/1d378e21-60e7-469f-9582-e58cd2e8086e-kube-api-access-sfqgq\") pod \"1d378e21-60e7-469f-9582-e58cd2e8086e\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") "
Sep 10 16:52:00.743463 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.743445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-tmp-dir\") pod \"1d378e21-60e7-469f-9582-e58cd2e8086e\" (UID: \"1d378e21-60e7-469f-9582-e58cd2e8086e\") "
Sep 10 16:52:00.743776 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.743597    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d378e21-60e7-469f-9582-e58cd2e8086e" (UID: "1d378e21-60e7-469f-9582-e58cd2e8086e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:52:00.743776 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.743659    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-home" (OuterVolumeSpecName: "home") pod "1d378e21-60e7-469f-9582-e58cd2e8086e" (UID: "1d378e21-60e7-469f-9582-e58cd2e8086e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:52:00.743776 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.743749    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-model-cache\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:52:00.743776 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.743764    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-home\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:52:00.743776 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.743762    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d378e21-60e7-469f-9582-e58cd2e8086e" (UID: "1d378e21-60e7-469f-9582-e58cd2e8086e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:52:00.745545 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.745513    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-dshm" (OuterVolumeSpecName: "dshm") pod "1d378e21-60e7-469f-9582-e58cd2e8086e" (UID: "1d378e21-60e7-469f-9582-e58cd2e8086e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:52:00.745645 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.745624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d378e21-60e7-469f-9582-e58cd2e8086e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d378e21-60e7-469f-9582-e58cd2e8086e" (UID: "1d378e21-60e7-469f-9582-e58cd2e8086e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:52:00.745685 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.745672    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d378e21-60e7-469f-9582-e58cd2e8086e-kube-api-access-sfqgq" (OuterVolumeSpecName: "kube-api-access-sfqgq") pod "1d378e21-60e7-469f-9582-e58cd2e8086e" (UID: "1d378e21-60e7-469f-9582-e58cd2e8086e"). InnerVolumeSpecName "kube-api-access-sfqgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:52:00.844161 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.844095    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d378e21-60e7-469f-9582-e58cd2e8086e-tls-certs\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:52:00.844161 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.844117    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfqgq\" (UniqueName: \"kubernetes.io/projected/1d378e21-60e7-469f-9582-e58cd2e8086e-kube-api-access-sfqgq\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:52:00.844161 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.844125    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-tmp-dir\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:52:00.844161 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:00.844133    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d378e21-60e7-469f-9582-e58cd2e8086e-dshm\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 10 16:52:01.496364 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:01.496330    2580 generic.go:356] "Generic (PLEG): container finished" podID="1d378e21-60e7-469f-9582-e58cd2e8086e" containerID="66bf39727b6ddc6912eb68b89bba11a3257b56425bd0469e3e11722d575e74a1" exitCode=0
Sep 10 16:52:01.496855 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:01.496376    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"
Sep 10 16:52:01.496855 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:01.496410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq" event={"ID":"1d378e21-60e7-469f-9582-e58cd2e8086e","Type":"ContainerDied","Data":"66bf39727b6ddc6912eb68b89bba11a3257b56425bd0469e3e11722d575e74a1"}
Sep 10 16:52:01.496855 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:01.496446    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq" event={"ID":"1d378e21-60e7-469f-9582-e58cd2e8086e","Type":"ContainerDied","Data":"f75457ad2ce5d0ef75e4e802899dc77a2fdb7a70e6789d72b084197f5a59805b"}
Sep 10 16:52:01.496855 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:01.496462    2580 scope.go:117] "RemoveContainer" containerID="66bf39727b6ddc6912eb68b89bba11a3257b56425bd0469e3e11722d575e74a1"
Sep 10 16:52:01.505469 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:01.505453    2580 scope.go:117] "RemoveContainer" containerID="66bf39727b6ddc6912eb68b89bba11a3257b56425bd0469e3e11722d575e74a1"
Sep 10 16:52:01.505743 ip-10-0-138-74 kubenswrapper[2580]: E0910 16:52:01.505725    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66bf39727b6ddc6912eb68b89bba11a3257b56425bd0469e3e11722d575e74a1\": container with ID starting with 66bf39727b6ddc6912eb68b89bba11a3257b56425bd0469e3e11722d575e74a1 not found: ID does not exist" containerID="66bf39727b6ddc6912eb68b89bba11a3257b56425bd0469e3e11722d575e74a1"
Sep 10 16:52:01.505811 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:01.505751    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66bf39727b6ddc6912eb68b89bba11a3257b56425bd0469e3e11722d575e74a1"} err="failed to get container status \"66bf39727b6ddc6912eb68b89bba11a3257b56425bd0469e3e11722d575e74a1\": rpc error: code = NotFound desc = could not find container \"66bf39727b6ddc6912eb68b89bba11a3257b56425bd0469e3e11722d575e74a1\": container with ID starting with 66bf39727b6ddc6912eb68b89bba11a3257b56425bd0469e3e11722d575e74a1 not found: ID does not exist"
Sep 10 16:52:01.517082 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:01.517051    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"]
Sep 10 16:52:01.520842 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:01.520818    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-3d45f766-kserve-8469c54599-hfwwq"]
Sep 10 16:52:02.254149 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:02.254116    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d378e21-60e7-469f-9582-e58cd2e8086e" path="/var/lib/kubelet/pods/1d378e21-60e7-469f-9582-e58cd2e8086e/volumes"
Sep 10 16:52:02.254511 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:02.254497    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95e5302c-3f82-4988-9995-56ab70a0f24c" path="/var/lib/kubelet/pods/95e5302c-3f82-4988-9995-56ab70a0f24c/volumes"
Sep 10 16:52:13.865479 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.865447    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"]
Sep 10 16:52:13.865873 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.865761    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95e5302c-3f82-4988-9995-56ab70a0f24c" containerName="main"
Sep 10 16:52:13.865873 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.865772    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="95e5302c-3f82-4988-9995-56ab70a0f24c" containerName="main"
Sep 10 16:52:13.865873 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.865792    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d378e21-60e7-469f-9582-e58cd2e8086e" containerName="main"
Sep 10 16:52:13.865873 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.865799    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d378e21-60e7-469f-9582-e58cd2e8086e" containerName="main"
Sep 10 16:52:13.865873 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.865848    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d378e21-60e7-469f-9582-e58cd2e8086e" containerName="main"
Sep 10 16:52:13.865873 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.865856    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="95e5302c-3f82-4988-9995-56ab70a0f24c" containerName="main"
Sep 10 16:52:13.869894 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.869875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:13.873048 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.873024    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5q8xx\""
Sep 10 16:52:13.873048 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.873049    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 16:52:13.873296 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.873027    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 16:52:13.873296 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.873101    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 16:52:13.873296 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.873027    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:52:13.878173 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.878153    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"]
Sep 10 16:52:13.950350 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.950324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5szds\" (UniqueName: \"kubernetes.io/projected/d82a4aba-1bfa-4a27-ada8-7d114e3600d0-kube-api-access-5szds\") pod \"maas-api-ddc7f4579-pcldm\" (UID: \"d82a4aba-1bfa-4a27-ada8-7d114e3600d0\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:13.950508 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.950362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82a4aba-1bfa-4a27-ada8-7d114e3600d0-maas-api-tls\") pod \"maas-api-ddc7f4579-pcldm\" (UID: \"d82a4aba-1bfa-4a27-ada8-7d114e3600d0\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:13.950508 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.950449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82a4aba-1bfa-4a27-ada8-7d114e3600d0-openshift-service-ca\") pod \"maas-api-ddc7f4579-pcldm\" (UID: \"d82a4aba-1bfa-4a27-ada8-7d114e3600d0\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:13.950508 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:13.950487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82a4aba-1bfa-4a27-ada8-7d114e3600d0-metrics-certs\") pod \"maas-api-ddc7f4579-pcldm\" (UID: \"d82a4aba-1bfa-4a27-ada8-7d114e3600d0\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:14.051640 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:14.051608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5szds\" (UniqueName: \"kubernetes.io/projected/d82a4aba-1bfa-4a27-ada8-7d114e3600d0-kube-api-access-5szds\") pod \"maas-api-ddc7f4579-pcldm\" (UID: \"d82a4aba-1bfa-4a27-ada8-7d114e3600d0\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:14.051773 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:14.051649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82a4aba-1bfa-4a27-ada8-7d114e3600d0-maas-api-tls\") pod \"maas-api-ddc7f4579-pcldm\" (UID: \"d82a4aba-1bfa-4a27-ada8-7d114e3600d0\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:14.051773 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:14.051711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82a4aba-1bfa-4a27-ada8-7d114e3600d0-openshift-service-ca\") pod \"maas-api-ddc7f4579-pcldm\" (UID: \"d82a4aba-1bfa-4a27-ada8-7d114e3600d0\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:14.051773 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:14.051728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82a4aba-1bfa-4a27-ada8-7d114e3600d0-metrics-certs\") pod \"maas-api-ddc7f4579-pcldm\" (UID: \"d82a4aba-1bfa-4a27-ada8-7d114e3600d0\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:14.052403 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:14.052376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82a4aba-1bfa-4a27-ada8-7d114e3600d0-openshift-service-ca\") pod \"maas-api-ddc7f4579-pcldm\" (UID: \"d82a4aba-1bfa-4a27-ada8-7d114e3600d0\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:14.054140 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:14.054117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82a4aba-1bfa-4a27-ada8-7d114e3600d0-metrics-certs\") pod \"maas-api-ddc7f4579-pcldm\" (UID: \"d82a4aba-1bfa-4a27-ada8-7d114e3600d0\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:14.054253 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:14.054142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82a4aba-1bfa-4a27-ada8-7d114e3600d0-maas-api-tls\") pod \"maas-api-ddc7f4579-pcldm\" (UID: \"d82a4aba-1bfa-4a27-ada8-7d114e3600d0\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:14.060373 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:14.060342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5szds\" (UniqueName: \"kubernetes.io/projected/d82a4aba-1bfa-4a27-ada8-7d114e3600d0-kube-api-access-5szds\") pod \"maas-api-ddc7f4579-pcldm\" (UID: \"d82a4aba-1bfa-4a27-ada8-7d114e3600d0\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:14.180556 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:14.180456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:14.453802 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:14.453775    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"]
Sep 10 16:52:14.456144 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:52:14.456115    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82a4aba_1bfa_4a27_ada8_7d114e3600d0.slice/crio-385fdd9d5f4c353956e75f1c7efbe84bfa838586c88e6166904569034e3be3e0 WatchSource:0}: Error finding container 385fdd9d5f4c353956e75f1c7efbe84bfa838586c88e6166904569034e3be3e0: Status 404 returned error can't find the container with id 385fdd9d5f4c353956e75f1c7efbe84bfa838586c88e6166904569034e3be3e0
Sep 10 16:52:14.457463 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:14.457446    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 16:52:14.552432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:14.552398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm" event={"ID":"d82a4aba-1bfa-4a27-ada8-7d114e3600d0","Type":"ContainerStarted","Data":"385fdd9d5f4c353956e75f1c7efbe84bfa838586c88e6166904569034e3be3e0"}
Sep 10 16:52:17.567220 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:17.567181    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm" event={"ID":"d82a4aba-1bfa-4a27-ada8-7d114e3600d0","Type":"ContainerStarted","Data":"00dd499693c263f2d9c16e0bdc88f7a1cf9fb46d5444ab0a29d3e8c1d8ba7421"}
Sep 10 16:52:17.567666 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:17.567328    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:17.586311 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:17.586264    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm" podStartSLOduration=2.379831763 podStartE2EDuration="4.586250845s" totalImagesPullingTime="2.206419082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:52:17.583956823 +0000 UTC m=+2095.784999235" watchObservedRunningTime="2026-09-10 16:52:17.586250845 +0000 UTC m=+2095.787293256"
Sep 10 16:52:23.577658 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:23.577627    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-ddc7f4579-pcldm"
Sep 10 16:52:36.407928 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:36.407878    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d68676b9c-6k2dt_b0969416-fcf6-4877-87f0-fe847a133951/authorino/0.log"
Sep 10 16:52:41.996707 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:41.996675    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7rtlr_11e4d234-8c3e-4ad2-8931-99af2824b387/server/0.log"
Sep 10 16:52:42.112105 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:42.112075    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-n9ffr_14f7f3f3-6711-4eee-9ef1-34a8100a357f/manager/0.log"
Sep 10 16:52:42.223282 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:42.223256    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57b77968d9-5ht8c_81302c0c-3dae-4657-9a6d-af3540e0e8ac/copy-manifests/0.log"
Sep 10 16:52:42.231181 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:42.231157    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57b77968d9-5ht8c_81302c0c-3dae-4657-9a6d-af3540e0e8ac/manager/0.log"
Sep 10 16:52:42.808337 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:42.808306    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-ddc7f4579-pcldm_d82a4aba-1bfa-4a27-ada8-7d114e3600d0/maas-api/0.log"
Sep 10 16:52:43.033840 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:43.033812    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817630-7nbjs_55dc7d49-aa0a-4869-8bb6-6d6bb94d1314/cleanup/0.log"
Sep 10 16:52:43.140752 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:43.140669    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817645-c2v9j_3022ce00-cee6-4698-acd3-faf1d42619be/cleanup/0.log"
Sep 10 16:52:43.988514 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:43.988481    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d68676b9c-6k2dt_b0969416-fcf6-4877-87f0-fe847a133951/authorino/0.log"
Sep 10 16:52:45.345384 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:45.345354    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cfcc4b4c4-xdjsr_d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f/kube-auth-proxy/0.log"
Sep 10 16:52:45.694768 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:45.694696    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mr4zw_d6605008-787d-480d-b24b-5e343dfe6d4d/payload-processing/0.log"
Sep 10 16:52:46.140212 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:46.140184    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvsz_5b390380-da50-4968-ae32-e69721f82605/main/0.log"
Sep 10 16:52:46.359857 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:46.359804    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-wc74c_0f411e02-ca8f-4fad-b849-6008e9785c3c/main/0.log"
Sep 10 16:52:46.795301 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:46.795269    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-2t5d4_b99d13c8-5ee1-4ea5-a38a-e2a297d8cced/main/0.log"
Sep 10 16:52:55.188460 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:55.188430    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dc7jq_25606248-47cf-4de0-aab7-48d1123e9901/global-pull-secret-syncer/0.log"
Sep 10 16:52:55.361254 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:55.361226    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5vf27_cc5e8ba0-2f3b-49df-b7e0-221a9ff548b8/konnectivity-agent/0.log"
Sep 10 16:52:55.437101 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:52:55.437072    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-74.ec2.internal_18ee5e215f5e8ba4676df80e0d119df6/haproxy/0.log"
Sep 10 16:53:00.203792 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:00.203758    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d68676b9c-6k2dt_b0969416-fcf6-4877-87f0-fe847a133951/authorino/0.log"
Sep 10 16:53:02.372736 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:02.372706    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4jjk_8b1b78b7-342c-4e54-bc3c-2771f3ff25cc/node-exporter/0.log"
Sep 10 16:53:02.393632 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:02.393603    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4jjk_8b1b78b7-342c-4e54-bc3c-2771f3ff25cc/kube-rbac-proxy/0.log"
Sep 10 16:53:02.414579 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:02.414559    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4jjk_8b1b78b7-342c-4e54-bc3c-2771f3ff25cc/init-textfile/0.log"
Sep 10 16:53:03.085192 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:03.085159    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-r2g6m_e5e1f877-d651-464b-9047-3c64e36a8f0d/prometheus-operator-admission-webhook/0.log"
Sep 10 16:53:04.632424 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:04.632396    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-m5m78_9cb81b47-97bd-4561-85c3-054dbcf875b5/networking-console-plugin/0.log"
Sep 10 16:53:05.653874 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.653830    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"]
Sep 10 16:53:05.657327 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.657301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.659333 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.659311    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbsrw\"/\"kube-root-ca.crt\""
Sep 10 16:53:05.659456 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.659427    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rbsrw\"/\"default-dockercfg-8nc6g\""
Sep 10 16:53:05.660011 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.659994    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbsrw\"/\"openshift-service-ca.crt\""
Sep 10 16:53:05.665518 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.665495    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"]
Sep 10 16:53:05.778461 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.778427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d08a7f00-c500-44ed-b5bf-e4671607a403-podres\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.778694 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.778470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbktr\" (UniqueName: \"kubernetes.io/projected/d08a7f00-c500-44ed-b5bf-e4671607a403-kube-api-access-mbktr\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.778694 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.778557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d08a7f00-c500-44ed-b5bf-e4671607a403-lib-modules\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.778694 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.778600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d08a7f00-c500-44ed-b5bf-e4671607a403-proc\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.778694 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.778632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d08a7f00-c500-44ed-b5bf-e4671607a403-sys\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.879510 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.879475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbktr\" (UniqueName: \"kubernetes.io/projected/d08a7f00-c500-44ed-b5bf-e4671607a403-kube-api-access-mbktr\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.879510 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.879514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d08a7f00-c500-44ed-b5bf-e4671607a403-lib-modules\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.879811 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.879551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d08a7f00-c500-44ed-b5bf-e4671607a403-proc\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.879811 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.879580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d08a7f00-c500-44ed-b5bf-e4671607a403-sys\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.879811 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.879657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d08a7f00-c500-44ed-b5bf-e4671607a403-podres\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.879811 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.879689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d08a7f00-c500-44ed-b5bf-e4671607a403-lib-modules\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.879811 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.879701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d08a7f00-c500-44ed-b5bf-e4671607a403-proc\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.879811 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.879751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d08a7f00-c500-44ed-b5bf-e4671607a403-sys\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.879811 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.879770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d08a7f00-c500-44ed-b5bf-e4671607a403-podres\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.887679 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.887635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbktr\" (UniqueName: \"kubernetes.io/projected/d08a7f00-c500-44ed-b5bf-e4671607a403-kube-api-access-mbktr\") pod \"perf-node-gather-daemonset-l8l5g\" (UID: \"d08a7f00-c500-44ed-b5bf-e4671607a403\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:05.967888 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:05.967796    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:06.237703 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:06.237674    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"]
Sep 10 16:53:06.240486 ip-10-0-138-74 kubenswrapper[2580]: W0910 16:53:06.240450    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd08a7f00_c500_44ed_b5bf_e4671607a403.slice/crio-7b3230a31cf08ea7086f38fc9184eca4c75e2fe65856a60b14ffee8c4cf24acf WatchSource:0}: Error finding container 7b3230a31cf08ea7086f38fc9184eca4c75e2fe65856a60b14ffee8c4cf24acf: Status 404 returned error can't find the container with id 7b3230a31cf08ea7086f38fc9184eca4c75e2fe65856a60b14ffee8c4cf24acf
Sep 10 16:53:06.783259 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:06.783217    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g" event={"ID":"d08a7f00-c500-44ed-b5bf-e4671607a403","Type":"ContainerStarted","Data":"8b15811511965fbabb2e015162a1d5fc8a5d6365bc468bb8f94cced6082a5a3c"}
Sep 10 16:53:06.783725 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:06.783282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g" event={"ID":"d08a7f00-c500-44ed-b5bf-e4671607a403","Type":"ContainerStarted","Data":"7b3230a31cf08ea7086f38fc9184eca4c75e2fe65856a60b14ffee8c4cf24acf"}
Sep 10 16:53:06.783725 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:06.783362    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:06.800217 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:06.800166    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g" podStartSLOduration=1.8001492479999999 podStartE2EDuration="1.800149248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:53:06.799848092 +0000 UTC m=+2145.000890499" watchObservedRunningTime="2026-09-10 16:53:06.800149248 +0000 UTC m=+2145.001191660"
Sep 10 16:53:07.616308 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:07.616279    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ptvp7_3b5cd229-4191-436f-bae3-841284e8f8f2/dns/0.log"
Sep 10 16:53:07.638384 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:07.638361    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ptvp7_3b5cd229-4191-436f-bae3-841284e8f8f2/kube-rbac-proxy/0.log"
Sep 10 16:53:07.835802 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:07.835769    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pgclc_2fa5acfb-ee22-43bb-9ca6-64284bb4dcb8/dns-node-resolver/0.log"
Sep 10 16:53:08.383357 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:08.383322    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5c6859976-hjrp4_ba733a9e-f54d-4aab-b83d-53d91ed0c306/registry/0.log"
Sep 10 16:53:08.403557 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:08.403520    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2zpjw_d95286cd-ebcd-47c2-9a95-d51f5e62576e/node-ca/0.log"
Sep 10 16:53:09.772465 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:09.772439    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cfcc4b4c4-xdjsr_d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f/kube-auth-proxy/0.log"
Sep 10 16:53:09.884677 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:09.884645    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mr4zw_d6605008-787d-480d-b24b-5e343dfe6d4d/payload-processing/0.log"
Sep 10 16:53:10.820842 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:10.820806    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2bbf6_fc083413-78ac-40c4-b86c-16a0aec316f2/serve-healthcheck-canary/0.log"
Sep 10 16:53:11.421618 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:11.421590    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k4nxn_afe1b19f-f425-4126-87e5-5a5c0be3b4bd/kube-rbac-proxy/0.log"
Sep 10 16:53:11.446184 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:11.446164    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k4nxn_afe1b19f-f425-4126-87e5-5a5c0be3b4bd/exporter/0.log"
Sep 10 16:53:11.468160 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:11.468142    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k4nxn_afe1b19f-f425-4126-87e5-5a5c0be3b4bd/extractor/0.log"
Sep 10 16:53:12.796249 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:12.796222    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-l8l5g"
Sep 10 16:53:14.013432 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:14.013406    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7rtlr_11e4d234-8c3e-4ad2-8931-99af2824b387/server/0.log"
Sep 10 16:53:14.095126 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:14.095092    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-n9ffr_14f7f3f3-6711-4eee-9ef1-34a8100a357f/manager/0.log"
Sep 10 16:53:14.123678 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:14.123643    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57b77968d9-5ht8c_81302c0c-3dae-4657-9a6d-af3540e0e8ac/manager/0.log"
Sep 10 16:53:14.156306 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:14.156281    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57b77968d9-5ht8c_81302c0c-3dae-4657-9a6d-af3540e0e8ac/copy-manifests/0.log"
Sep 10 16:53:15.633751 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:15.633669    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-575575bffc-xp7vh_cc1812ef-b56a-4e01-828b-c58850fad789/manager/0.log"
Sep 10 16:53:16.258431 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:16.258406    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cfcc4b4c4-xdjsr_d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f/kube-auth-proxy/0.log"
Sep 10 16:53:16.343806 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:16.343775    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mr4zw_d6605008-787d-480d-b24b-5e343dfe6d4d/payload-processing/0.log"
Sep 10 16:53:18.280516 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:18.280486    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cfcc4b4c4-xdjsr_d5d2caa6-f481-4ea7-99c9-b1aa91ddc42f/kube-auth-proxy/0.log"
Sep 10 16:53:18.370721 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:18.370696    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mr4zw_d6605008-787d-480d-b24b-5e343dfe6d4d/payload-processing/0.log"
Sep 10 16:53:22.464458 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:22.464417    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-72nfz_f719995b-ab3b-4e8c-a7b0-d055cd79caf5/kube-multus/0.log"
Sep 10 16:53:22.847289 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:22.847253    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4wn7_87b01bad-5f40-4c30-aa70-bfd96d239873/kube-multus-additional-cni-plugins/0.log"
Sep 10 16:53:22.873348 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:22.873316    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4wn7_87b01bad-5f40-4c30-aa70-bfd96d239873/egress-router-binary-copy/0.log"
Sep 10 16:53:22.894076 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:22.894050    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4wn7_87b01bad-5f40-4c30-aa70-bfd96d239873/cni-plugins/0.log"
Sep 10 16:53:22.914172 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:22.914143    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4wn7_87b01bad-5f40-4c30-aa70-bfd96d239873/bond-cni-plugin/0.log"
Sep 10 16:53:22.932472 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:22.932442    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4wn7_87b01bad-5f40-4c30-aa70-bfd96d239873/routeoverride-cni/0.log"
Sep 10 16:53:22.953843 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:22.953818    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4wn7_87b01bad-5f40-4c30-aa70-bfd96d239873/whereabouts-cni-bincopy/0.log"
Sep 10 16:53:22.972656 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:22.972630    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4wn7_87b01bad-5f40-4c30-aa70-bfd96d239873/whereabouts-cni/0.log"
Sep 10 16:53:23.123056 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:23.122975    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lnx4z_0b2115e2-15e0-4fe4-9bfd-1158f59a0c10/network-metrics-daemon/0.log"
Sep 10 16:53:23.143170 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:23.143142    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lnx4z_0b2115e2-15e0-4fe4-9bfd-1158f59a0c10/kube-rbac-proxy/0.log"
Sep 10 16:53:24.172983 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:24.172953    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbjsw_4d3fba85-1986-4255-befb-90e94d9aa2e4/ovn-controller/0.log"
Sep 10 16:53:24.189330 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:24.189307    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbjsw_4d3fba85-1986-4255-befb-90e94d9aa2e4/ovn-acl-logging/0.log"
Sep 10 16:53:24.210660 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:24.210630    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbjsw_4d3fba85-1986-4255-befb-90e94d9aa2e4/kube-rbac-proxy-node/0.log"
Sep 10 16:53:24.231022 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:24.230990    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbjsw_4d3fba85-1986-4255-befb-90e94d9aa2e4/kube-rbac-proxy-ovn-metrics/0.log"
Sep 10 16:53:24.250637 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:24.250612    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbjsw_4d3fba85-1986-4255-befb-90e94d9aa2e4/northd/0.log"
Sep 10 16:53:24.271729 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:24.271698    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbjsw_4d3fba85-1986-4255-befb-90e94d9aa2e4/nbdb/0.log"
Sep 10 16:53:24.291962 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:24.291938    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbjsw_4d3fba85-1986-4255-befb-90e94d9aa2e4/sbdb/0.log"
Sep 10 16:53:24.342990 ip-10-0-138-74 kubenswrapper[2580]: I0910 16:53:24.342956    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbjsw_4d3fba85-1986-4255-befb-90e94d9aa2e4/ovnkube-controller/0.log"