Sep 08 09:59:27.066304 ip-10-0-128-212 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 09:59:27.066317 ip-10-0-128-212 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 09:59:27.066325 ip-10-0-128-212 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 09:59:27.066470 ip-10-0-128-212 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 09:59:37.315282 ip-10-0-128-212 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 08 09:59:37.315298 ip-10-0-128-212 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f2626254230d42959fbb9f78813fae57 --
Sep 08 10:01:18.729032 ip-10-0-128-212 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 10:01:19.169743 ip-10-0-128-212 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 10:01:19.169743 ip-10-0-128-212 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 10:01:19.169743 ip-10-0-128-212 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 10:01:19.169743 ip-10-0-128-212 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 10:01:19.169743 ip-10-0-128-212 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 10:01:19.170932 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.170856    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 10:01:19.175141 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175125    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 10:01:19.175141 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175139    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 10:01:19.175141 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175143    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 10:01:19.175141 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175146    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175150    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175154    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175157    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175160    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175163    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175165    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175168    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175170    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175173    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175176    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175179    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175182    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175186    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175190    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175193    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175196    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175199    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175202    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175205    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 10:01:19.175286 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175207    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175210    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175213    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175215    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175218    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175220    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175223    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175225    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175228    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175231    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175233    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175236    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175239    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175244    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175248    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175251    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175255    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175258    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175261    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 10:01:19.175767 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175264    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175267    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175269    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175272    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175274    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175277    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175281    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175283    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175286    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175289    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175291    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175294    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175297    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175299    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175303    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175307    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175310    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175313    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 10:01:19.176339 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175316    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175319    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175322    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175325    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175327    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175331    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175335    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175338    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175341    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175345    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175348    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175351    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175354    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175357    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175359    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175362    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175364    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175367    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175370    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175372    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 10:01:19.177041 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175375    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175378    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175381    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175384    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175387    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175389    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175392    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175394    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175397    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175399    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175402    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175404    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175407    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175409    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175412    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175414    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175417    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175420    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.175422    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176482    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 10:01:19.177531 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176496    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176501    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176506    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176510    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176514    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176524    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176528    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176533    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176536    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176541    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176546    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176552    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176558    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176562    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176568    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176572    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176577    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176586    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176590    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 10:01:19.178034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176594    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176599    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176603    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176607    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176612    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176617    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176621    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176625    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176629    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176633    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176638    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176647    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176651    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176655    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176659    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176663    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176668    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176672    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176676    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176680    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 10:01:19.178509 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176684    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176688    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176693    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176696    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176705    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176709    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176713    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176717    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176722    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176727    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176731    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176736    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176742    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176747    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176753    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176763    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176767    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176772    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176776    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 10:01:19.179013 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176796    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176800    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176805    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176810    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176814    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176819    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176823    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176827    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176831    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176839    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176844    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176848    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176852    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176856    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176861    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176868    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176873    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176877    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176882    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176887    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 10:01:19.179475 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176891    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176900    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176906    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176910    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176918    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176924    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176929    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176933    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176936    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176940    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176945    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176949    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176958    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176962    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176966    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176970    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176974    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176978    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176982    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 10:01:19.179978 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.176986    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177560    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177571    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177575    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177578    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177582    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177585    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177588    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177591    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177594    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177597    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177599    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177602    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177605    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177608    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177611    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177614    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177617    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177619    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177622    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 10:01:19.180463 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177625    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177628    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177631    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177634    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177637    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177639    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177642    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177644    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177647    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177650    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177652    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177655    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177657    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177660    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177663    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177666    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177669    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177671    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177674    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177678    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 10:01:19.180969 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177680    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177683    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177685    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177688    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177690    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177693    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177696    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177700    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177704    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177707    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177710    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177713    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177716    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177718    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177721    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177724    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177726    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177730    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177733    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 10:01:19.182180 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177736    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177739    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177742    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177745    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177748    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177750    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177753    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177756    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177759    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177761    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177764    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177767    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177769    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177772    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177774    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177793    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177796    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177799    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177801    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177804    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 10:01:19.182676 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177806    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177809    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177812    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177814    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177817    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177820    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177823    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177825    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177829    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177832    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177835    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177837    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177840    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177843    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177845    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177849    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177853    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177856    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177859    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177862    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 10:01:19.183190 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.177865    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.177953    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.177993    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178000    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178004    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178008    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178011    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178015    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178020    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178023    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178026    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178029    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178033    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178036    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178039    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178042    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178046    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178049    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178052    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178057    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178060    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178063    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178066    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178069    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 10:01:19.183680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178073    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178075    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178079    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178082    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178088    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178091    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178094    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178097    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178100    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178104    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178107    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178110    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178113    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178116    2559 flags.go:64] FLAG: --enable-server="true"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178119    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178123    2559 flags.go:64] FLAG: --event-burst="100"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178126    2559 flags.go:64] FLAG: --event-qps="50"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178129    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178132    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178135    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178138    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178141    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178144    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178147    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178150    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 10:01:19.184314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178153    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178157    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178160    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178163    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178166    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178168    2559 flags.go:64] FLAG: --feature-gates=""
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178172    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178175    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178178    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178181    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178184    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178188    2559 flags.go:64] FLAG: --help="false"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178191    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178195    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178197    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178200    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178204    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178207    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178210    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178213    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178215    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178218    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178221    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178224    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 10:01:19.184947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178227    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178230    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178232    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178235    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178238    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178241    2559 flags.go:64] FLAG: --lock-file=""
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178243    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178247    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178250    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178259    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178263    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178266    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178269    2559 flags.go:64] FLAG: --logging-format="text"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178272    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178275    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178278    2559 flags.go:64] FLAG: --manifest-url=""
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178281    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178285    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178288    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178293    2559 flags.go:64] FLAG: --max-pods="110"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178296    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178299    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178301    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178304    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178307    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 10:01:19.185531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178310    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178313    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178320    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178323    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178326    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178329    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178331    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178337    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178340    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178343    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178346    2559 flags.go:64] FLAG: --port="10250"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178349    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178351    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-036b7bd0f18f36b4e"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178355    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178358    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178361    2559 flags.go:64] FLAG: --register-node="true"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178363    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178367    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178371    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178374    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178377    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178381    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178384    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178386    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178389    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 10:01:19.186190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178392    2559 flags.go:64] FLAG: --runonce="false"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178395    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178399    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178402    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178405    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178408    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178411    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178414    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178417    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178420    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178424    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178427    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178429    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178432    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178435    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178438    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178443    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178446    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178448    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178452    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178455    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178459    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178462    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178465    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178468    2559 flags.go:64] FLAG: --v="2"
Sep 08 10:01:19.186777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178472    2559 flags.go:64] FLAG: --version="false"
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178476    2559 flags.go:64] FLAG: --vmodule=""
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178480    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.178483    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178594    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178598    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178601    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178604    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178607    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178611    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178613    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178616    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178618    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178621    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178624    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178626    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178629    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178631    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178634    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178637    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 10:01:19.187438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178639    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178642    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178644    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178647    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178649    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178652    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178654    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178657    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178661    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178664    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178666    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178670    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178674    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178677    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178680    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178683    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178686    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178688    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178691    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 10:01:19.188033 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178695    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178698    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178701    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178704    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178707    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178710    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178713    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178716    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178718    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178721    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178723    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178726    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178728    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178731    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178734    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178736    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178739    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178741    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178744    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178747    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 10:01:19.188496 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178749    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178753    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178756    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178759    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178762    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178764    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178767    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178769    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178772    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178775    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178794    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178797    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178799    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178802    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178805    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178808    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178810    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178813    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178815    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 10:01:19.189004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178818    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178821    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178824    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178826    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178829    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178831    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178834    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178836    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178839    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178843    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178846    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178849    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178852    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178855    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178859    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178862    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178865    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178867    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178870    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 10:01:19.189466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178872    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178875    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178877    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178880    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178883    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.178886    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.179595    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.187061    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.187075    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.187926    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.187933    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188015    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188021    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188024    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188027    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 10:01:19.189932 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188030    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188035    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188039    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188042    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188045    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188048    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188050    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188053    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188056    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188059    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188061    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188064    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188066    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188069    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188072    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188074    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188077    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188080    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188082    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 10:01:19.190304 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188085    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188088    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188090    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188093    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188095    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188098    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188102    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188105    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188107    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188110    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188112    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188115    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188118    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188120    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188123    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188125    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188128    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188130    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188133    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188135    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 10:01:19.190768 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188138    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188141    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188143    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188146    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188149    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188151    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188153    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188156    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188159    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188161    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188163    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188166    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188168    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188171    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188173    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188176    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188178    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188181    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188184    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188187    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 10:01:19.191278 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188190    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188193    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188195    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188198    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188201    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188203    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188206    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188208    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188211    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188213    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188216    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188218    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188220    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188223    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188226    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188229    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188231    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188234    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188236    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188238    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 10:01:19.191811 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188243    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188246    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188249    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188252    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188255    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188258    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188261    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188264    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188267    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188271    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188274    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188277    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188281    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188284    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188287    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 10:01:19.192301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188290    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 10:01:19.192810 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.188295    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 10:01:19.192810 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188418    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 10:01:19.192810 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188423    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 10:01:19.192810 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188426    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 10:01:19.192810 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188429    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 10:01:19.192810 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188432    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 10:01:19.192810 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188434    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 10:01:19.192810 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188437    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 10:01:19.192810 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188441    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 10:01:19.192810 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188445    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 10:01:19.192810 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188449    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 10:01:19.192810 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188451    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 10:01:19.192810 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188454    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 10:01:19.192810 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188457    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188459    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188462    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188464    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188467    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188469    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188472    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188474    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188477    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188480    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188482    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188484    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188487    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188490    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188493    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188496    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188500    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188502    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188505    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188508    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 10:01:19.193158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188511    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188515    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188518    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188521    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188524    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188527    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188530    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188532    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188536    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188538    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188541    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188544    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188546    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188549    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188552    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188554    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188557    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188560    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188562    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 10:01:19.193683 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188565    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188567    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188570    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188572    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188575    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188578    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188581    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188584    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188586    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188589    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188593    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188596    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188599    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188602    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188605    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188608    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188611    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188614    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188617    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 10:01:19.194158 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188620    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188622    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188625    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188628    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188631    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188633    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188636    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188638    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188641    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188643    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188646    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188649    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188651    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188654    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188656    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188659    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188662    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188664    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188667    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188670    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 10:01:19.194622 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188672    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 10:01:19.195215 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188675    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 10:01:19.195215 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188677    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 10:01:19.195215 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188680    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 10:01:19.195215 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188683    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 10:01:19.195215 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188685    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 10:01:19.195215 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188688    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 10:01:19.195215 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188690    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 10:01:19.195215 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:19.188693    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 10:01:19.195215 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.188697    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 10:01:19.195215 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.188809    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 10:01:19.195215 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.191266    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 10:01:19.195215 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.192399    2559 server.go:1019] "Starting client certificate rotation"
Sep 08 10:01:19.195215 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.192513    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 10:01:19.195215 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.192605    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 10:01:19.195576 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.195205    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 10:01:19.198501 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.198480    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 10:01:19.209416 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.209396    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 08 10:01:19.214633 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.214613    2559 log.go:25] "Validated CRI v1 image API"
Sep 08 10:01:19.215510 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.215486    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 10:01:19.219534 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.219514    2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bf3a3273-eaef-4fae-89ee-596bdda9e57b:/dev/nvme0n1p3 e375e0db-fb8b-4142-a4db-4e9f28e17453:/dev/nvme0n1p4]
Sep 08 10:01:19.219605 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.219531    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 10:01:19.224234 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.224219    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 10:01:19.226255 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.226153    2559 manager.go:216] Machine: {Timestamp:2026-09-08 10:01:19.224439068 +0000 UTC m=+0.370125362 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092143 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec293bdc6c28d09681f1a745fae1600c SystemUUID:ec293bdc-6c28-d096-81f1-a745fae1600c BootID:f2626254-230d-4295-9fbb-9f78813fae57 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5d:66:25:1d:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5d:66:25:1d:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:15:dd:c5:ab:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 10:01:19.226255 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.226251    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 10:01:19.226360 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.226329    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 10:01:19.229726 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.229702    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 10:01:19.229866 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.229728    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-212.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 10:01:19.231282 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.231272    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 10:01:19.231316 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.231284    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 10:01:19.231316 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.231291    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 10:01:19.232101 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.232090    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 10:01:19.234077 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.234067    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 10:01:19.237876 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.237864    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 10:01:19.237991 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.237982    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 10:01:19.238740 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.238728    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 10:01:19.238811 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.238744    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 10:01:19.238811 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.238762    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 10:01:19.238811 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.238770    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 08 10:01:19.238811 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.238791    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 10:01:19.239135 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.239124    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 10:01:19.239189 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.239145    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 10:01:19.241495 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.241480    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 10:01:19.244239 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.244212    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 10:01:19.244323 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.244250    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 10:01:19.244437 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.244425    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 10:01:19.244499 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.244444    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 10:01:19.244499 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.244453    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 10:01:19.244499 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.244462    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 10:01:19.244499 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.244473    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 10:01:19.244499 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.244482    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 10:01:19.245002 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.244991    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 10:01:19.245052 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.245003    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 10:01:19.245052 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.245014    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 10:01:19.245052 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.245023    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 10:01:19.245052 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.245036    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 10:01:19.245052 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.245048    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 10:01:19.246878 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.246865    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 10:01:19.246945 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.246880    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 10:01:19.248550 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.248528    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-212.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 10:01:19.248623 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.248525    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 10:01:19.250589 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.250576    2559 server.go:1268] "Started kubelet"
Sep 08 10:01:19.250690 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.250669    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 10:01:19.251084 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.251025    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 10:01:19.251162 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.251105    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 10:01:19.251434 ip-10-0-128-212 systemd[1]: Started Kubernetes Kubelet.
Sep 08 10:01:19.252586 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.252566    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 10:01:19.254455 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.254435    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 10:01:19.256096 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.256072    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 10:01:19.258630 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.258612    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-212.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 10:01:19.260067 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.258689    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-212.ec2.internal.18d35038300986dd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-212.ec2.internal,UID:ip-10-0-128-212.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-212.ec2.internal,},FirstTimestamp:2026-09-08 10:01:19.250540253 +0000 UTC m=+0.396226521,LastTimestamp:2026-09-08 10:01:19.250540253 +0000 UTC m=+0.396226521,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-212.ec2.internal,}"
Sep 08 10:01:19.260299 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.260281    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 10:01:19.261267 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.261249    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 10:01:19.261438 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.261416    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:19.261925 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.261904    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 10:01:19.261925 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.261904    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 10:01:19.262046 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.261936    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 10:01:19.262105 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.262051    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 10:01:19.262105 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.262058    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 10:01:19.262855 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.262825    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 10:01:19.262855 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.262840    2559 factory.go:55] Registering systemd factory
Sep 08 10:01:19.262855 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.262848    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 08 10:01:19.263096 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.263060    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-212.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 10:01:19.263096 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.263072    2559 factory.go:153] Registering CRI-O factory
Sep 08 10:01:19.263096 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.263092    2559 factory.go:223] Registration of the crio container factory successfully
Sep 08 10:01:19.263256 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.263105    2559 factory.go:103] Registering Raw factory
Sep 08 10:01:19.263256 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.263115    2559 manager.go:1196] Started watching for new ooms in manager
Sep 08 10:01:19.263256 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.263200    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 10:01:19.263521 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.263508    2559 manager.go:349] Starting recovery of all containers
Sep 08 10:01:19.273567 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.273544    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7czkp"
Sep 08 10:01:19.275002 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.274988    2559 manager.go:354] Recovery completed
Sep 08 10:01:19.280336 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.280322    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 10:01:19.280438 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.280424    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7czkp"
Sep 08 10:01:19.282417 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.282320    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 10:01:19.282479 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.282429    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 10:01:19.282479 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.282440    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasSufficientPID"
Sep 08 10:01:19.283013 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.282983    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 10:01:19.283013 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.282996    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 10:01:19.283013 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.282998    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 10:01:19.283013 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.283009    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 10:01:19.283836 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.283822    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 10:01:19.283899 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.283850    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 10:01:19.283899 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.283865    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasSufficientPID"
Sep 08 10:01:19.284141 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.284129    2559 policy_none.go:49] "None policy: Start"
Sep 08 10:01:19.284175 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.284145    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 10:01:19.284175 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.284159    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 10:01:19.284238 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.284114    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-212.ec2.internal.18d3503831efea7f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-212.ec2.internal,UID:ip-10-0-128-212.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-212.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-212.ec2.internal,},FirstTimestamp:2026-09-08 10:01:19.282416255 +0000 UTC m=+0.428102522,LastTimestamp:2026-09-08 10:01:19.282416255 +0000 UTC m=+0.428102522,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-212.ec2.internal,}"
Sep 08 10:01:19.285083 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.285067    2559 policy_none.go:47] "Start"
Sep 08 10:01:19.310327 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.310313    2559 manager.go:337] "Starting Device Plugin manager"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.310345    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.310354    2559 server.go:87] "Starting device plugin registration server"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.310639    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.310653    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.310743    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.310853    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.310864    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.311334    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.311373    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.324169    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.325297    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.325350    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.325373    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.325444    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.327468    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.411352    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.412203    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.412228    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.412240    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasSufficientPID"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.412261    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.420928    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.420945    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-212.ec2.internal\": node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.425668    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-212.ec2.internal"]
Sep 08 10:01:19.426240 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.425723    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 10:01:19.427064 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.426540    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 10:01:19.427064 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.426563    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 10:01:19.427064 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.426577    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasSufficientPID"
Sep 08 10:01:19.427753 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.427741    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 10:01:19.427910 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.427893    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.427960 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.427928    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 10:01:19.428426 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.428408    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 10:01:19.428426 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.428418    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 10:01:19.428567 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.428436    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 10:01:19.428567 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.428444    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 10:01:19.428567 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.428474    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasSufficientPID"
Sep 08 10:01:19.428567 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.428447    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasSufficientPID"
Sep 08 10:01:19.429571 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.429559    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.429623 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.429589    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 10:01:19.430214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.430200    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 10:01:19.430278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.430222    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 10:01:19.430278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.430231    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeHasSufficientPID"
Sep 08 10:01:19.445405 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.445389    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:19.461417 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.461396    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-212.ec2.internal\" not found" node="ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.463355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.463332    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/052f4ba795232d6489bc9a1bde394dc3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-212.ec2.internal\" (UID: \"052f4ba795232d6489bc9a1bde394dc3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.463458 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.463366    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6009be1324bc0532adc60e45c833edd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal\" (UID: \"6009be1324bc0532adc60e45c833edd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.463458 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.463392    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6009be1324bc0532adc60e45c833edd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal\" (UID: \"6009be1324bc0532adc60e45c833edd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.465817 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.465801    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-212.ec2.internal\" not found" node="ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.546448 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.546421    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:19.563764 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.563738    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6009be1324bc0532adc60e45c833edd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal\" (UID: \"6009be1324bc0532adc60e45c833edd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.563764 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.563690    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6009be1324bc0532adc60e45c833edd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal\" (UID: \"6009be1324bc0532adc60e45c833edd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.563906 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.563819    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6009be1324bc0532adc60e45c833edd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal\" (UID: \"6009be1324bc0532adc60e45c833edd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.563906 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.563839    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/052f4ba795232d6489bc9a1bde394dc3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-212.ec2.internal\" (UID: \"052f4ba795232d6489bc9a1bde394dc3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.563906 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.563878    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/052f4ba795232d6489bc9a1bde394dc3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-212.ec2.internal\" (UID: \"052f4ba795232d6489bc9a1bde394dc3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.563906 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.563898    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6009be1324bc0532adc60e45c833edd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal\" (UID: \"6009be1324bc0532adc60e45c833edd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.646875 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.646849    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:19.747700 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.747613    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:19.765046 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.765024    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.770603 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:19.770583    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:19.847718 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.847694    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:19.948261 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:19.948238    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:20.048754 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:20.048692    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:20.149370 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:20.149347    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:20.191769 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:20.191751    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 10:01:20.192249 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:20.191891    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 10:01:21.554721 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:21.554698    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:21.587621 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:21.587593    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 09:56:20 +0000 UTC" deadline="2028-03-23 00:05:06.174899056 +0000 UTC"
Sep 08 10:01:21.587621 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:21.587616    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13478h3m44.58728581s"
Sep 08 10:01:21.639706 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:21.639658    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 10:01:21.655415 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:21.655397    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:21.755822 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:21.755796    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:21.764629 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:21.764611    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 10:01:21.856194 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:21.856171    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:21.956741 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:21.956692    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-212.ec2.internal\" not found"
Sep 08 10:01:22.033278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.033257    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 10:01:22.066290 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.066272    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:22.080223 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.080209    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 10:01:22.081361 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.081348    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-212.ec2.internal"
Sep 08 10:01:22.087531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.087518    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 10:01:22.324826 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:22.324802    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052f4ba795232d6489bc9a1bde394dc3.slice/crio-103952766d4c27cab1d2a382f5fcaa53be739cdf92cc68935296a2f762d48360 WatchSource:0}: Error finding container 103952766d4c27cab1d2a382f5fcaa53be739cdf92cc68935296a2f762d48360: Status 404 returned error can't find the container with id 103952766d4c27cab1d2a382f5fcaa53be739cdf92cc68935296a2f762d48360
Sep 08 10:01:22.330277 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.330261    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 10:01:22.544456 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.544393    2559 apiserver.go:52] "Watching apiserver"
Sep 08 10:01:22.555097 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.555077    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 10:01:22.555434 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.555415    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-8sj86","openshift-multus/multus-additional-cni-plugins-4jxq2","kube-system/konnectivity-agent-w8khh","openshift-cluster-node-tuning-operator/tuned-fw22r","openshift-multus/network-metrics-daemon-2pgcf","openshift-network-diagnostics/network-check-target-pgp7f","openshift-network-operator/iptables-alerter-gbj76","openshift-ovn-kubernetes/ovnkube-node-6dtdm","kube-system/global-pull-secret-syncer-9h7fv","kube-system/kube-apiserver-proxy-ip-10-0-128-212.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm","openshift-dns/node-resolver-5lzjn","openshift-image-registry/node-ca-hp9jt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal"]
Sep 08 10:01:22.558012 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.557999    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.559058 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.559042    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hp9jt"
Sep 08 10:01:22.560061 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.560048    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w8khh"
Sep 08 10:01:22.560382 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.560362    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 10:01:22.560479 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.560404    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 10:01:22.560536 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.560519    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n9jht\""
Sep 08 10:01:22.560625 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.560604    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 10:01:22.561212 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.561193    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 10:01:22.561318 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.561224    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5x2pn\""
Sep 08 10:01:22.561318 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.561197    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 10:01:22.561318 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.561264    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.561318 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.561191    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 10:01:22.562124 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.562105    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x28rs\""
Sep 08 10:01:22.562218 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.562129    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 10:01:22.562363 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.562349    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 10:01:22.562575 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.562563    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:22.562652 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:22.562619    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:22.563284 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.563266    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c2zdm\""
Sep 08 10:01:22.563668 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.563650    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 10:01:22.563765 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.563699    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 10:01:22.563765 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.563728    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5lzjn"
Sep 08 10:01:22.563888 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.563794    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 10:01:22.563888 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.563837    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 10:01:22.565232 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.565003    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gbj76"
Sep 08 10:01:22.565548 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.565534    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 10:01:22.565762 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.565747    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fxg4d\""
Sep 08 10:01:22.565865 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.565849    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 10:01:22.565916 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.565885    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 10:01:22.566522 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.566510    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.567482 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.567466    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 10:01:22.567570 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.567498    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qs7sj\""
Sep 08 10:01:22.567570 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.567513    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 10:01:22.567570 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.567546    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 10:01:22.567860 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.567845    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:22.567904 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:22.567894    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:22.568549 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.568535    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 10:01:22.568908 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.568894    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 10:01:22.568983 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.568892    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2dftb\""
Sep 08 10:01:22.569164 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.569143    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.570952 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.570797    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:22.570952 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:22.570868    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:22.571493 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.571457    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 10:01:22.571579 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.571517    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vzkdv\""
Sep 08 10:01:22.571669 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.571654    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 10:01:22.571773 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.571759    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 10:01:22.571833 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.571820    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 10:01:22.571913 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.571898    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 10:01:22.572499 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.572479    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.572788 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.572772    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 10:01:22.574872 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.574852    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ng9q2\""
Sep 08 10:01:22.574872 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.574865    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 10:01:22.575016 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.574887    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 10:01:22.575528 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.575504    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 10:01:22.576312 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576292    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a972fa38-95b8-4809-934a-bdf008f50db5-etc-tuned\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.576381 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576324    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvfs\" (UniqueName: \"kubernetes.io/projected/a972fa38-95b8-4809-934a-bdf008f50db5-kube-api-access-zfvfs\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.576381 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:22.576463 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576399    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-multus-cni-dir\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.576506 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576460    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-hostroot\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.576506 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576482    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-sysctl-conf\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.576506 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576497    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-var-lib-kubelet\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.576620 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576518    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.576620 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576566    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-run-k8s-cni-cncf-io\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.576620 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576594    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-run-netns\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.576730 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27334e80-03d1-4b30-a744-8b6ea5eaf69f-iptables-alerter-script\") pod \"iptables-alerter-gbj76\" (UID: \"27334e80-03d1-4b30-a744-8b6ea5eaf69f\") " pod="openshift-network-operator/iptables-alerter-gbj76"
Sep 08 10:01:22.576730 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576662    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27334e80-03d1-4b30-a744-8b6ea5eaf69f-host-slash\") pod \"iptables-alerter-gbj76\" (UID: \"27334e80-03d1-4b30-a744-8b6ea5eaf69f\") " pod="openshift-network-operator/iptables-alerter-gbj76"
Sep 08 10:01:22.576730 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576686    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs\") pod \"network-metrics-daemon-2pgcf\" (UID: \"a8a40235-67e9-4b28-ad0c-97751e2afe10\") " pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:22.576730 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576700    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff849db0-791c-405b-bbfc-868f1b7288d2-hosts-file\") pod \"node-resolver-5lzjn\" (UID: \"ff849db0-791c-405b-bbfc-868f1b7288d2\") " pod="openshift-dns/node-resolver-5lzjn"
Sep 08 10:01:22.576886 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576737    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff849db0-791c-405b-bbfc-868f1b7288d2-tmp-dir\") pod \"node-resolver-5lzjn\" (UID: \"ff849db0-791c-405b-bbfc-868f1b7288d2\") " pod="openshift-dns/node-resolver-5lzjn"
Sep 08 10:01:22.576886 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576760    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a972fa38-95b8-4809-934a-bdf008f50db5-tmp\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.576886 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-multus-socket-dir-parent\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.576886 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fdf41754-0a6d-4c9c-93c5-240108b5c9db-multus-daemon-config\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.576886 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-etc-kubernetes\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.576886 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbdwj\" (UniqueName: \"kubernetes.io/projected/ff849db0-791c-405b-bbfc-868f1b7288d2-kube-api-access-nbdwj\") pod \"node-resolver-5lzjn\" (UID: \"ff849db0-791c-405b-bbfc-868f1b7288d2\") " pod="openshift-dns/node-resolver-5lzjn"
Sep 08 10:01:22.577279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-kubernetes\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.577279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576960    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-run\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.577279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.576984    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-sys\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.577279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577005    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81df3687-05a6-4b77-ac67-2f8c6626e906-kubelet-config\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:22.577279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577077    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-system-cni-dir\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.577279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577101    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-var-lib-cni-bin\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.577279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577147    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-var-lib-cni-multus\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.577279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577172    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2719b26a-ad31-4018-a46c-927706ebc26e-serviceca\") pod \"node-ca-hp9jt\" (UID: \"2719b26a-ad31-4018-a46c-927706ebc26e\") " pod="openshift-image-registry/node-ca-hp9jt"
Sep 08 10:01:22.577279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577193    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-systemd\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.577279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577214    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-multus-conf-dir\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.577279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577234    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-run-multus-certs\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.577279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s6wc\" (UniqueName: \"kubernetes.io/projected/fdf41754-0a6d-4c9c-93c5-240108b5c9db-kube-api-access-6s6wc\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577322    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmxhb\" (UniqueName: \"kubernetes.io/projected/27334e80-03d1-4b30-a744-8b6ea5eaf69f-kube-api-access-gmxhb\") pod \"iptables-alerter-gbj76\" (UID: \"27334e80-03d1-4b30-a744-8b6ea5eaf69f\") " pod="openshift-network-operator/iptables-alerter-gbj76"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577348    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81df3687-05a6-4b77-ac67-2f8c6626e906-dbus\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577371    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-socket-dir\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577396    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fdf41754-0a6d-4c9c-93c5-240108b5c9db-cni-binary-copy\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577431    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-var-lib-kubelet\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577456    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcknn\" (UniqueName: \"kubernetes.io/projected/a8a40235-67e9-4b28-ad0c-97751e2afe10-kube-api-access-gcknn\") pod \"network-metrics-daemon-2pgcf\" (UID: \"a8a40235-67e9-4b28-ad0c-97751e2afe10\") " pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577570    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-sysctl-d\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577593    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-lib-modules\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577621    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-device-dir\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577649    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-etc-selinux\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577671    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-sys-fs\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577688    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-cnibin\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577704    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2719b26a-ad31-4018-a46c-927706ebc26e-host\") pod \"node-ca-hp9jt\" (UID: \"2719b26a-ad31-4018-a46c-927706ebc26e\") " pod="openshift-image-registry/node-ca-hp9jt"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577719    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg5jw\" (UniqueName: \"kubernetes.io/projected/2719b26a-ad31-4018-a46c-927706ebc26e-kube-api-access-dg5jw\") pod \"node-ca-hp9jt\" (UID: \"2719b26a-ad31-4018-a46c-927706ebc26e\") " pod="openshift-image-registry/node-ca-hp9jt"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577733    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-sysconfig\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.577787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577753    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-registration-dir\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.578277 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577769    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvbgs\" (UniqueName: \"kubernetes.io/projected/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-kube-api-access-cvbgs\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.578277 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577783    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62ec76ef-d924-4dbb-b66b-735e295ddc0c-agent-certs\") pod \"konnectivity-agent-w8khh\" (UID: \"62ec76ef-d924-4dbb-b66b-735e295ddc0c\") " pod="kube-system/konnectivity-agent-w8khh"
Sep 08 10:01:22.578277 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62ec76ef-d924-4dbb-b66b-735e295ddc0c-konnectivity-ca\") pod \"konnectivity-agent-w8khh\" (UID: \"62ec76ef-d924-4dbb-b66b-735e295ddc0c\") " pod="kube-system/konnectivity-agent-w8khh"
Sep 08 10:01:22.578277 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577811    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-modprobe-d\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.578277 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.577829    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-host\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.601304 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.601285    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7vt5x"
Sep 08 10:01:22.609118 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.609099    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7vt5x"
Sep 08 10:01:22.633541 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.633508    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-212.ec2.internal" event={"ID":"052f4ba795232d6489bc9a1bde394dc3","Type":"ContainerStarted","Data":"103952766d4c27cab1d2a382f5fcaa53be739cdf92cc68935296a2f762d48360"}
Sep 08 10:01:22.667637 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.667619    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 10:01:22.678617 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-var-lib-kubelet\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.678681 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678621    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-slash\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.678681 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678637    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-var-lib-openvswitch\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.678681 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678652    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-cni-binary-copy\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.678681 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678671    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.678872 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678697    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-sysctl-d\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.678872 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678713    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-var-lib-kubelet\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.678872 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678731    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-lib-modules\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.678872 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-etc-selinux\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.678872 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-cnibin\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.678872 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678846    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-sysctl-d\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.678872 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678851    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-etc-selinux\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.678872 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-run-k8s-cni-cncf-io\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678885    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-cnibin\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678886    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27334e80-03d1-4b30-a744-8b6ea5eaf69f-iptables-alerter-script\") pod \"iptables-alerter-gbj76\" (UID: \"27334e80-03d1-4b30-a744-8b6ea5eaf69f\") " pod="openshift-network-operator/iptables-alerter-gbj76"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678902    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-lib-modules\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-sysconfig\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678934    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-registration-dir\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678944    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-run-k8s-cni-cncf-io\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678950    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvbgs\" (UniqueName: \"kubernetes.io/projected/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-kube-api-access-cvbgs\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678972    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-sysconfig\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-registration-dir\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.678999    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62ec76ef-d924-4dbb-b66b-735e295ddc0c-agent-certs\") pod \"konnectivity-agent-w8khh\" (UID: \"62ec76ef-d924-4dbb-b66b-735e295ddc0c\") " pod="kube-system/konnectivity-agent-w8khh"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62ec76ef-d924-4dbb-b66b-735e295ddc0c-konnectivity-ca\") pod \"konnectivity-agent-w8khh\" (UID: \"62ec76ef-d924-4dbb-b66b-735e295ddc0c\") " pod="kube-system/konnectivity-agent-w8khh"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679087    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679112    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvfs\" (UniqueName: \"kubernetes.io/projected/a972fa38-95b8-4809-934a-bdf008f50db5-kube-api-access-zfvfs\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679147    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679176    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-run-netns\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679204    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-system-cni-dir\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.679231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679228    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45fxz\" (UniqueName: \"kubernetes.io/projected/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-kube-api-access-45fxz\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:22.679252    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679269    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-sysctl-conf\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-var-lib-kubelet\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679395    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-sysctl-conf\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:22.679413    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret podName:81df3687-05a6-4b77-ac67-2f8c6626e906 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:23.179368031 +0000 UTC m=+3.020592409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret") pod "global-pull-secret-syncer-9h7fv" (UID: "81df3687-05a6-4b77-ac67-2f8c6626e906") : object "kube-system"/"original-pull-secret" not registered
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679441    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27334e80-03d1-4b30-a744-8b6ea5eaf69f-iptables-alerter-script\") pod \"iptables-alerter-gbj76\" (UID: \"27334e80-03d1-4b30-a744-8b6ea5eaf69f\") " pod="openshift-network-operator/iptables-alerter-gbj76"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679460    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679465    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-var-lib-kubelet\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679493    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-systemd-units\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679556    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679547    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-ovn-node-metrics-cert\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679550    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679603    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvwdd\" (UniqueName: \"kubernetes.io/projected/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-kube-api-access-wvwdd\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679646    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-multus-socket-dir-parent\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679689    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf4lj\" (UniqueName: \"kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj\") pod \"network-check-target-pgp7f\" (UID: \"d8cad0d3-0f69-491d-a066-2718f066d04d\") " pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:22.679929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679710    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbdwj\" (UniqueName: \"kubernetes.io/projected/ff849db0-791c-405b-bbfc-868f1b7288d2-kube-api-access-nbdwj\") pod \"node-resolver-5lzjn\" (UID: \"ff849db0-791c-405b-bbfc-868f1b7288d2\") " pod="openshift-dns/node-resolver-5lzjn"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679722    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-multus-socket-dir-parent\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679730    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-kubernetes\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-run\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679771    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-var-lib-cni-bin\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679789    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-run\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679792    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-var-lib-cni-multus\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679818    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-var-lib-cni-bin\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679818    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-multus-conf-dir\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679850    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-multus-conf-dir\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-kubernetes\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679854    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-var-lib-cni-multus\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679895    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-log-socket\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679934    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-env-overrides\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679959    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2719b26a-ad31-4018-a46c-927706ebc26e-serviceca\") pod \"node-ca-hp9jt\" (UID: \"2719b26a-ad31-4018-a46c-927706ebc26e\") " pod="openshift-image-registry/node-ca-hp9jt"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.679985    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-run-multus-certs\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-run-ovn\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680031    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-cni-netd\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.680739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680043    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-run-multus-certs\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680052    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-ovnkube-script-lib\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680079    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-cnibin\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81df3687-05a6-4b77-ac67-2f8c6626e906-dbus\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680126    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fdf41754-0a6d-4c9c-93c5-240108b5c9db-cni-binary-copy\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680147    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-kubelet\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680174    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcknn\" (UniqueName: \"kubernetes.io/projected/a8a40235-67e9-4b28-ad0c-97751e2afe10-kube-api-access-gcknn\") pod \"network-metrics-daemon-2pgcf\" (UID: \"a8a40235-67e9-4b28-ad0c-97751e2afe10\") " pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680196    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-device-dir\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680219    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-sys-fs\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-run-netns\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680260    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81df3687-05a6-4b77-ac67-2f8c6626e906-dbus\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680287    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-run-openvswitch\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680346    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-sys-fs\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680353    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-host-run-netns\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fdf41754-0a6d-4c9c-93c5-240108b5c9db-multus-daemon-config\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2719b26a-ad31-4018-a46c-927706ebc26e-host\") pod \"node-ca-hp9jt\" (UID: \"2719b26a-ad31-4018-a46c-927706ebc26e\") " pod="openshift-image-registry/node-ca-hp9jt"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680397    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2719b26a-ad31-4018-a46c-927706ebc26e-serviceca\") pod \"node-ca-hp9jt\" (UID: \"2719b26a-ad31-4018-a46c-927706ebc26e\") " pod="openshift-image-registry/node-ca-hp9jt"
Sep 08 10:01:22.681653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680406    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-device-dir\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg5jw\" (UniqueName: \"kubernetes.io/projected/2719b26a-ad31-4018-a46c-927706ebc26e-kube-api-access-dg5jw\") pod \"node-ca-hp9jt\" (UID: \"2719b26a-ad31-4018-a46c-927706ebc26e\") " pod="openshift-image-registry/node-ca-hp9jt"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680450    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2719b26a-ad31-4018-a46c-927706ebc26e-host\") pod \"node-ca-hp9jt\" (UID: \"2719b26a-ad31-4018-a46c-927706ebc26e\") " pod="openshift-image-registry/node-ca-hp9jt"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-multus-cni-dir\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680502    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-etc-openvswitch\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680505    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62ec76ef-d924-4dbb-b66b-735e295ddc0c-konnectivity-ca\") pod \"konnectivity-agent-w8khh\" (UID: \"62ec76ef-d924-4dbb-b66b-735e295ddc0c\") " pod="kube-system/konnectivity-agent-w8khh"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-modprobe-d\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680550    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-host\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680556    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-multus-cni-dir\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680593    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a972fa38-95b8-4809-934a-bdf008f50db5-etc-tuned\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680599    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-host\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-modprobe-d\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680621    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-hostroot\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680644    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27334e80-03d1-4b30-a744-8b6ea5eaf69f-host-slash\") pod \"iptables-alerter-gbj76\" (UID: \"27334e80-03d1-4b30-a744-8b6ea5eaf69f\") " pod="openshift-network-operator/iptables-alerter-gbj76"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680660    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-hostroot\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680691    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680715    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-ovnkube-config\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680742    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-node-log\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.682411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680695    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27334e80-03d1-4b30-a744-8b6ea5eaf69f-host-slash\") pod \"iptables-alerter-gbj76\" (UID: \"27334e80-03d1-4b30-a744-8b6ea5eaf69f\") " pod="openshift-network-operator/iptables-alerter-gbj76"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680762    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs\") pod \"network-metrics-daemon-2pgcf\" (UID: \"a8a40235-67e9-4b28-ad0c-97751e2afe10\") " pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680797    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff849db0-791c-405b-bbfc-868f1b7288d2-hosts-file\") pod \"node-resolver-5lzjn\" (UID: \"ff849db0-791c-405b-bbfc-868f1b7288d2\") " pod="openshift-dns/node-resolver-5lzjn"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fdf41754-0a6d-4c9c-93c5-240108b5c9db-multus-daemon-config\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff849db0-791c-405b-bbfc-868f1b7288d2-tmp-dir\") pod \"node-resolver-5lzjn\" (UID: \"ff849db0-791c-405b-bbfc-868f1b7288d2\") " pod="openshift-dns/node-resolver-5lzjn"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680875    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a972fa38-95b8-4809-934a-bdf008f50db5-tmp\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:22.680894    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680899    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-etc-kubernetes\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-sys\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:22.680937    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs podName:a8a40235-67e9-4b28-ad0c-97751e2afe10 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:23.180922525 +0000 UTC m=+3.022146901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs") pod "network-metrics-daemon-2pgcf" (UID: "a8a40235-67e9-4b28-ad0c-97751e2afe10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680967    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81df3687-05a6-4b77-ac67-2f8c6626e906-kubelet-config\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680978    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-sys\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681014    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-system-cni-dir\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681041    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-run-systemd\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681067    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-cni-bin\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.680893    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff849db0-791c-405b-bbfc-868f1b7288d2-hosts-file\") pod \"node-resolver-5lzjn\" (UID: \"ff849db0-791c-405b-bbfc-868f1b7288d2\") " pod="openshift-dns/node-resolver-5lzjn"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681080    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff849db0-791c-405b-bbfc-868f1b7288d2-tmp-dir\") pod \"node-resolver-5lzjn\" (UID: \"ff849db0-791c-405b-bbfc-868f1b7288d2\") " pod="openshift-dns/node-resolver-5lzjn"
Sep 08 10:01:22.683214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681115    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.683767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681070    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81df3687-05a6-4b77-ac67-2f8c6626e906-kubelet-config\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:22.683767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681137    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-system-cni-dir\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.683767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681017    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdf41754-0a6d-4c9c-93c5-240108b5c9db-etc-kubernetes\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.683767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-systemd\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.683767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681278    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s6wc\" (UniqueName: \"kubernetes.io/projected/fdf41754-0a6d-4c9c-93c5-240108b5c9db-kube-api-access-6s6wc\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.683767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681285    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a972fa38-95b8-4809-934a-bdf008f50db5-etc-systemd\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.683767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681303    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmxhb\" (UniqueName: \"kubernetes.io/projected/27334e80-03d1-4b30-a744-8b6ea5eaf69f-kube-api-access-gmxhb\") pod \"iptables-alerter-gbj76\" (UID: \"27334e80-03d1-4b30-a744-8b6ea5eaf69f\") " pod="openshift-network-operator/iptables-alerter-gbj76"
Sep 08 10:01:22.683767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681354    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-socket-dir\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.683767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681487    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fdf41754-0a6d-4c9c-93c5-240108b5c9db-cni-binary-copy\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.683767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.681522    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-socket-dir\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.683767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.683426    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a972fa38-95b8-4809-934a-bdf008f50db5-tmp\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.683767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.683489    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a972fa38-95b8-4809-934a-bdf008f50db5-etc-tuned\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.683767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.683681    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62ec76ef-d924-4dbb-b66b-735e295ddc0c-agent-certs\") pod \"konnectivity-agent-w8khh\" (UID: \"62ec76ef-d924-4dbb-b66b-735e295ddc0c\") " pod="kube-system/konnectivity-agent-w8khh"
Sep 08 10:01:22.687520 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.687479    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvbgs\" (UniqueName: \"kubernetes.io/projected/dee1a539-f4cb-40e2-af51-a67bc1d9dccc-kube-api-access-cvbgs\") pod \"aws-ebs-csi-driver-node-4wjgm\" (UID: \"dee1a539-f4cb-40e2-af51-a67bc1d9dccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.687615 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.687593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbdwj\" (UniqueName: \"kubernetes.io/projected/ff849db0-791c-405b-bbfc-868f1b7288d2-kube-api-access-nbdwj\") pod \"node-resolver-5lzjn\" (UID: \"ff849db0-791c-405b-bbfc-868f1b7288d2\") " pod="openshift-dns/node-resolver-5lzjn"
Sep 08 10:01:22.687838 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.687816    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvfs\" (UniqueName: \"kubernetes.io/projected/a972fa38-95b8-4809-934a-bdf008f50db5-kube-api-access-zfvfs\") pod \"tuned-fw22r\" (UID: \"a972fa38-95b8-4809-934a-bdf008f50db5\") " pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.687915 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.687896    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg5jw\" (UniqueName: \"kubernetes.io/projected/2719b26a-ad31-4018-a46c-927706ebc26e-kube-api-access-dg5jw\") pod \"node-ca-hp9jt\" (UID: \"2719b26a-ad31-4018-a46c-927706ebc26e\") " pod="openshift-image-registry/node-ca-hp9jt"
Sep 08 10:01:22.688599 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.688583    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcknn\" (UniqueName: \"kubernetes.io/projected/a8a40235-67e9-4b28-ad0c-97751e2afe10-kube-api-access-gcknn\") pod \"network-metrics-daemon-2pgcf\" (UID: \"a8a40235-67e9-4b28-ad0c-97751e2afe10\") " pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:22.688973 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.688951    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmxhb\" (UniqueName: \"kubernetes.io/projected/27334e80-03d1-4b30-a744-8b6ea5eaf69f-kube-api-access-gmxhb\") pod \"iptables-alerter-gbj76\" (UID: \"27334e80-03d1-4b30-a744-8b6ea5eaf69f\") " pod="openshift-network-operator/iptables-alerter-gbj76"
Sep 08 10:01:22.689045 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.689024    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s6wc\" (UniqueName: \"kubernetes.io/projected/fdf41754-0a6d-4c9c-93c5-240108b5c9db-kube-api-access-6s6wc\") pod \"multus-8sj86\" (UID: \"fdf41754-0a6d-4c9c-93c5-240108b5c9db\") " pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.782518 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.782497    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-systemd-units\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.782649 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.782523    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-ovn-node-metrics-cert\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.782649 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.782540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvwdd\" (UniqueName: \"kubernetes.io/projected/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-kube-api-access-wvwdd\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.782649 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.782555    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.782649 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.782629    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-systemd-units\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.782814 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.782778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf4lj\" (UniqueName: \"kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj\") pod \"network-check-target-pgp7f\" (UID: \"d8cad0d3-0f69-491d-a066-2718f066d04d\") " pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:22.782814 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.782800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-log-socket\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.782905 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.782818    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-env-overrides\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.782905 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.782843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-run-ovn\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.782905 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.782891    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-run-ovn\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.782905 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.782900    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-log-socket\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783049 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.782997    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-cni-netd\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783049 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-ovnkube-script-lib\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783137 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783059    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-cnibin\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.783137 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-kubelet\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783137 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783089    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-cni-netd\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783137 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783116    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-run-openvswitch\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783137 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783128    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-cnibin\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.783362 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-etc-openvswitch\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783362 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783172    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-kubelet\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783362 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783172    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783362 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.783362 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783209    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-etc-openvswitch\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783362 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783209    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783362 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-ovnkube-config\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783362 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783262    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-run-openvswitch\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783362 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783271    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-node-log\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783362 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783312    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-run-systemd\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783362 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-cni-bin\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783416    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-node-log\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-env-overrides\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783498    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-run-systemd\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783502    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-slash\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783544    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783560    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-ovnkube-script-lib\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783568    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-slash\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783575    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-cni-bin\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-var-lib-openvswitch\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783613    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-var-lib-openvswitch\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783640    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-ovnkube-config\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-cni-binary-copy\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783711    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783735    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-run-netns\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.783779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783776    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-host-run-netns\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.784358 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-system-cni-dir\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.784358 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783828    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45fxz\" (UniqueName: \"kubernetes.io/projected/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-kube-api-access-45fxz\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.784358 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783872    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-system-cni-dir\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.784358 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.783901    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.784358 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.784018    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-cni-binary-copy\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.784358 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.784169    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.784721 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.784699    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-ovn-node-metrics-cert\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.788516 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:22.788501    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 10:01:22.788573 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:22.788521    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 10:01:22.788573 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:22.788534    2559 projected.go:196] Error preparing data for projected volume kube-api-access-hf4lj for pod openshift-network-diagnostics/network-check-target-pgp7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 10:01:22.788647 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:22.788590    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj podName:d8cad0d3-0f69-491d-a066-2718f066d04d nodeName:}" failed. No retries permitted until 2026-09-08 10:01:23.288578377 +0000 UTC m=+3.129802741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hf4lj" (UniqueName: "kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj") pod "network-check-target-pgp7f" (UID: "d8cad0d3-0f69-491d-a066-2718f066d04d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 10:01:22.790677 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.790660    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvwdd\" (UniqueName: \"kubernetes.io/projected/55e4fc3c-4c4e-4f2b-958e-961b61133b3c-kube-api-access-wvwdd\") pod \"ovnkube-node-6dtdm\" (UID: \"55e4fc3c-4c4e-4f2b-958e-961b61133b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.791235 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.791217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45fxz\" (UniqueName: \"kubernetes.io/projected/59f5177b-f1ce-4ef1-b7c2-752e58d5e27b-kube-api-access-45fxz\") pod \"multus-additional-cni-plugins-4jxq2\" (UID: \"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b\") " pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.842392 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:22.842371    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6009be1324bc0532adc60e45c833edd7.slice/crio-d49f32bebbe8cf9c32c43d03bdccd5de66888d76894f5dd912cd5158b32e87bf WatchSource:0}: Error finding container d49f32bebbe8cf9c32c43d03bdccd5de66888d76894f5dd912cd5158b32e87bf: Status 404 returned error can't find the container with id d49f32bebbe8cf9c32c43d03bdccd5de66888d76894f5dd912cd5158b32e87bf
Sep 08 10:01:22.868004 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.867985    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm"
Sep 08 10:01:22.873959 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.873941    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hp9jt"
Sep 08 10:01:22.875019 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:22.875000    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee1a539_f4cb_40e2_af51_a67bc1d9dccc.slice/crio-e6315c2ec69464af1e0877df269faf4f663e4f88521e1fda9a880af3830bc7a6 WatchSource:0}: Error finding container e6315c2ec69464af1e0877df269faf4f663e4f88521e1fda9a880af3830bc7a6: Status 404 returned error can't find the container with id e6315c2ec69464af1e0877df269faf4f663e4f88521e1fda9a880af3830bc7a6
Sep 08 10:01:22.880703 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:22.880683    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2719b26a_ad31_4018_a46c_927706ebc26e.slice/crio-ffc2bd9d40d52a053eec580406c49c1bb84c09607b4aad0c971422c7efddc047 WatchSource:0}: Error finding container ffc2bd9d40d52a053eec580406c49c1bb84c09607b4aad0c971422c7efddc047: Status 404 returned error can't find the container with id ffc2bd9d40d52a053eec580406c49c1bb84c09607b4aad0c971422c7efddc047
Sep 08 10:01:22.881740 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.881724    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w8khh"
Sep 08 10:01:22.886382 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.886367    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8sj86"
Sep 08 10:01:22.888477 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:22.888457    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ec76ef_d924_4dbb_b66b_735e295ddc0c.slice/crio-d14ae9fa069ecb55276f3e6532801ebb83ff57e4bd7cd44806a07ba05c68f3a0 WatchSource:0}: Error finding container d14ae9fa069ecb55276f3e6532801ebb83ff57e4bd7cd44806a07ba05c68f3a0: Status 404 returned error can't find the container with id d14ae9fa069ecb55276f3e6532801ebb83ff57e4bd7cd44806a07ba05c68f3a0
Sep 08 10:01:22.892174 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.892155    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5lzjn"
Sep 08 10:01:22.893338 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:22.893320    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf41754_0a6d_4c9c_93c5_240108b5c9db.slice/crio-5dc7828132d1ed3f4b76fefbde677e8da22d35aa1f0fcb3c2d39aeb2bca4c902 WatchSource:0}: Error finding container 5dc7828132d1ed3f4b76fefbde677e8da22d35aa1f0fcb3c2d39aeb2bca4c902: Status 404 returned error can't find the container with id 5dc7828132d1ed3f4b76fefbde677e8da22d35aa1f0fcb3c2d39aeb2bca4c902
Sep 08 10:01:22.898455 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.898437    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gbj76"
Sep 08 10:01:22.900261 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:22.900222    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff849db0_791c_405b_bbfc_868f1b7288d2.slice/crio-96bc0a100210e1b1b8b49e8fac052ae60b809f8e85af1975010b730afa4143fc WatchSource:0}: Error finding container 96bc0a100210e1b1b8b49e8fac052ae60b809f8e85af1975010b730afa4143fc: Status 404 returned error can't find the container with id 96bc0a100210e1b1b8b49e8fac052ae60b809f8e85af1975010b730afa4143fc
Sep 08 10:01:22.903806 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.903778    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fw22r"
Sep 08 10:01:22.905061 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:22.905044    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27334e80_03d1_4b30_a744_8b6ea5eaf69f.slice/crio-dc2d0bf134c987269265868b4806bc8220cf9d1ac0c43152b34d273b915fd8f0 WatchSource:0}: Error finding container dc2d0bf134c987269265868b4806bc8220cf9d1ac0c43152b34d273b915fd8f0: Status 404 returned error can't find the container with id dc2d0bf134c987269265868b4806bc8220cf9d1ac0c43152b34d273b915fd8f0
Sep 08 10:01:22.908345 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.908331    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:22.911561 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:22.911545    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda972fa38_95b8_4809_934a_bdf008f50db5.slice/crio-a2b5213fe7919e0e94de5a236fa2e753e2838e0c9fd8227dfeae954841fe6780 WatchSource:0}: Error finding container a2b5213fe7919e0e94de5a236fa2e753e2838e0c9fd8227dfeae954841fe6780: Status 404 returned error can't find the container with id a2b5213fe7919e0e94de5a236fa2e753e2838e0c9fd8227dfeae954841fe6780
Sep 08 10:01:22.912897 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:22.912874    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4jxq2"
Sep 08 10:01:22.915711 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:22.915694    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e4fc3c_4c4e_4f2b_958e_961b61133b3c.slice/crio-6fc631dd3db51ca75377255f27052a8c4f2176bc87d6864fc9885c147a031237 WatchSource:0}: Error finding container 6fc631dd3db51ca75377255f27052a8c4f2176bc87d6864fc9885c147a031237: Status 404 returned error can't find the container with id 6fc631dd3db51ca75377255f27052a8c4f2176bc87d6864fc9885c147a031237
Sep 08 10:01:22.920297 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:22.920145    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f5177b_f1ce_4ef1_b7c2_752e58d5e27b.slice/crio-ed3fd2f636f7e6603a842585028d2e83e32c363991d1a9e5b8c4f65ca9ff1397 WatchSource:0}: Error finding container ed3fd2f636f7e6603a842585028d2e83e32c363991d1a9e5b8c4f65ca9ff1397: Status 404 returned error can't find the container with id ed3fd2f636f7e6603a842585028d2e83e32c363991d1a9e5b8c4f65ca9ff1397
Sep 08 10:01:23.032302 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.032281    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 10:01:23.186684 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.186632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs\") pod \"network-metrics-daemon-2pgcf\" (UID: \"a8a40235-67e9-4b28-ad0c-97751e2afe10\") " pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:23.186684 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.186664    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:23.186805 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:23.186769    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 10:01:23.186837 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:23.186813    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 10:01:23.186837 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:23.186820    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs podName:a8a40235-67e9-4b28-ad0c-97751e2afe10 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:24.186807101 +0000 UTC m=+4.028031462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs") pod "network-metrics-daemon-2pgcf" (UID: "a8a40235-67e9-4b28-ad0c-97751e2afe10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 10:01:23.186904 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:23.186850    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret podName:81df3687-05a6-4b77-ac67-2f8c6626e906 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:24.186839967 +0000 UTC m=+4.028064345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret") pod "global-pull-secret-syncer-9h7fv" (UID: "81df3687-05a6-4b77-ac67-2f8c6626e906") : object "kube-system"/"original-pull-secret" not registered
Sep 08 10:01:23.388862 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.388836    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf4lj\" (UniqueName: \"kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj\") pod \"network-check-target-pgp7f\" (UID: \"d8cad0d3-0f69-491d-a066-2718f066d04d\") " pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:23.389014 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:23.388980    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 10:01:23.389014 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:23.388996    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 10:01:23.389014 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:23.389013    2559 projected.go:196] Error preparing data for projected volume kube-api-access-hf4lj for pod openshift-network-diagnostics/network-check-target-pgp7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 10:01:23.389112 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:23.389058    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj podName:d8cad0d3-0f69-491d-a066-2718f066d04d nodeName:}" failed. No retries permitted until 2026-09-08 10:01:24.389044649 +0000 UTC m=+4.230269022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hf4lj" (UniqueName: "kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj") pod "network-check-target-pgp7f" (UID: "d8cad0d3-0f69-491d-a066-2718f066d04d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 10:01:23.611005 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.610929    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 09:56:22 +0000 UTC" deadline="2028-03-06 15:08:43.236175235 +0000 UTC"
Sep 08 10:01:23.611005 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.610954    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13085h7m19.625224204s"
Sep 08 10:01:23.630329 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.630304    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:23.630425 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:23.630390    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:23.636306 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.636284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gbj76" event={"ID":"27334e80-03d1-4b30-a744-8b6ea5eaf69f","Type":"ContainerStarted","Data":"dc2d0bf134c987269265868b4806bc8220cf9d1ac0c43152b34d273b915fd8f0"}
Sep 08 10:01:23.637308 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.637291    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8sj86" event={"ID":"fdf41754-0a6d-4c9c-93c5-240108b5c9db","Type":"ContainerStarted","Data":"5dc7828132d1ed3f4b76fefbde677e8da22d35aa1f0fcb3c2d39aeb2bca4c902"}
Sep 08 10:01:23.638332 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.638303    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w8khh" event={"ID":"62ec76ef-d924-4dbb-b66b-735e295ddc0c","Type":"ContainerStarted","Data":"d14ae9fa069ecb55276f3e6532801ebb83ff57e4bd7cd44806a07ba05c68f3a0"}
Sep 08 10:01:23.639340 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.639322    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm" event={"ID":"dee1a539-f4cb-40e2-af51-a67bc1d9dccc","Type":"ContainerStarted","Data":"e6315c2ec69464af1e0877df269faf4f663e4f88521e1fda9a880af3830bc7a6"}
Sep 08 10:01:23.640310 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.640289    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal" event={"ID":"6009be1324bc0532adc60e45c833edd7","Type":"ContainerStarted","Data":"d49f32bebbe8cf9c32c43d03bdccd5de66888d76894f5dd912cd5158b32e87bf"}
Sep 08 10:01:23.641309 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.641286    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jxq2" event={"ID":"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b","Type":"ContainerStarted","Data":"ed3fd2f636f7e6603a842585028d2e83e32c363991d1a9e5b8c4f65ca9ff1397"}
Sep 08 10:01:23.642358 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.642341    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm" event={"ID":"55e4fc3c-4c4e-4f2b-958e-961b61133b3c","Type":"ContainerStarted","Data":"6fc631dd3db51ca75377255f27052a8c4f2176bc87d6864fc9885c147a031237"}
Sep 08 10:01:23.643362 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.643341    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fw22r" event={"ID":"a972fa38-95b8-4809-934a-bdf008f50db5","Type":"ContainerStarted","Data":"a2b5213fe7919e0e94de5a236fa2e753e2838e0c9fd8227dfeae954841fe6780"}
Sep 08 10:01:23.644341 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.644325    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5lzjn" event={"ID":"ff849db0-791c-405b-bbfc-868f1b7288d2","Type":"ContainerStarted","Data":"96bc0a100210e1b1b8b49e8fac052ae60b809f8e85af1975010b730afa4143fc"}
Sep 08 10:01:23.645251 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:23.645223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hp9jt" event={"ID":"2719b26a-ad31-4018-a46c-927706ebc26e","Type":"ContainerStarted","Data":"ffc2bd9d40d52a053eec580406c49c1bb84c09607b4aad0c971422c7efddc047"}
Sep 08 10:01:24.195861 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:24.195826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs\") pod \"network-metrics-daemon-2pgcf\" (UID: \"a8a40235-67e9-4b28-ad0c-97751e2afe10\") " pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:24.196045 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:24.195892    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:24.196045 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:24.196030    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 10:01:24.196151 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:24.196090    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret podName:81df3687-05a6-4b77-ac67-2f8c6626e906 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:26.196070817 +0000 UTC m=+6.037295197 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret") pod "global-pull-secret-syncer-9h7fv" (UID: "81df3687-05a6-4b77-ac67-2f8c6626e906") : object "kube-system"/"original-pull-secret" not registered
Sep 08 10:01:24.196505 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:24.196486    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 10:01:24.196600 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:24.196536    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs podName:a8a40235-67e9-4b28-ad0c-97751e2afe10 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:26.19652215 +0000 UTC m=+6.037746518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs") pod "network-metrics-daemon-2pgcf" (UID: "a8a40235-67e9-4b28-ad0c-97751e2afe10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 10:01:24.398541 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:24.398456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf4lj\" (UniqueName: \"kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj\") pod \"network-check-target-pgp7f\" (UID: \"d8cad0d3-0f69-491d-a066-2718f066d04d\") " pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:24.398694 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:24.398616    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 10:01:24.398694 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:24.398635    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 10:01:24.398694 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:24.398647    2559 projected.go:196] Error preparing data for projected volume kube-api-access-hf4lj for pod openshift-network-diagnostics/network-check-target-pgp7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 10:01:24.398792 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:24.398706    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj podName:d8cad0d3-0f69-491d-a066-2718f066d04d nodeName:}" failed. No retries permitted until 2026-09-08 10:01:26.398687766 +0000 UTC m=+6.239912135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hf4lj" (UniqueName: "kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj") pod "network-check-target-pgp7f" (UID: "d8cad0d3-0f69-491d-a066-2718f066d04d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 10:01:24.611685 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:24.611569    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 09:56:22 +0000 UTC" deadline="2028-02-15 19:15:42.578357549 +0000 UTC"
Sep 08 10:01:24.611685 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:24.611605    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12609h14m17.9667569s"
Sep 08 10:01:24.633990 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:24.633215    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:24.633990 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:24.633401    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:24.633990 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:24.633805    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:24.633990 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:24.633909    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:25.630621 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:25.630548    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:25.631007 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:25.630676    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:26.215515 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:26.215475    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:26.215692 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:26.215557    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs\") pod \"network-metrics-daemon-2pgcf\" (UID: \"a8a40235-67e9-4b28-ad0c-97751e2afe10\") " pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:26.215750 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:26.215736    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 10:01:26.215807 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:26.215798    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs podName:a8a40235-67e9-4b28-ad0c-97751e2afe10 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:30.215778398 +0000 UTC m=+10.057002765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs") pod "network-metrics-daemon-2pgcf" (UID: "a8a40235-67e9-4b28-ad0c-97751e2afe10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 10:01:26.216262 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:26.216204    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 10:01:26.216342 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:26.216268    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret podName:81df3687-05a6-4b77-ac67-2f8c6626e906 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:30.216236602 +0000 UTC m=+10.057460978 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret") pod "global-pull-secret-syncer-9h7fv" (UID: "81df3687-05a6-4b77-ac67-2f8c6626e906") : object "kube-system"/"original-pull-secret" not registered
Sep 08 10:01:26.417703 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:26.417666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf4lj\" (UniqueName: \"kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj\") pod \"network-check-target-pgp7f\" (UID: \"d8cad0d3-0f69-491d-a066-2718f066d04d\") " pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:26.417883 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:26.417841    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 10:01:26.417883 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:26.417865    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 10:01:26.417883 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:26.417879    2559 projected.go:196] Error preparing data for projected volume kube-api-access-hf4lj for pod openshift-network-diagnostics/network-check-target-pgp7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 10:01:26.418033 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:26.417939    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj podName:d8cad0d3-0f69-491d-a066-2718f066d04d nodeName:}" failed. No retries permitted until 2026-09-08 10:01:30.417921532 +0000 UTC m=+10.259145896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hf4lj" (UniqueName: "kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj") pod "network-check-target-pgp7f" (UID: "d8cad0d3-0f69-491d-a066-2718f066d04d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 10:01:26.631457 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:26.631367    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:26.631894 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:26.631505    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:26.633177 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:26.633154    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:26.633306 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:26.633285    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:27.630154 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:27.630119    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:27.630352 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:27.630258    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:28.630416 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:28.630387    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:28.630850 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:28.630531    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:28.630850 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:28.630543    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:28.630850 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:28.630661    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:29.631144 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:29.630609    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:29.631144 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:29.630731    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:30.249900 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:30.249775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:30.249900 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:30.249881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs\") pod \"network-metrics-daemon-2pgcf\" (UID: \"a8a40235-67e9-4b28-ad0c-97751e2afe10\") " pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:30.250129 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:30.250016    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 10:01:30.250129 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:30.250091    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs podName:a8a40235-67e9-4b28-ad0c-97751e2afe10 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:38.25007051 +0000 UTC m=+18.091294893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs") pod "network-metrics-daemon-2pgcf" (UID: "a8a40235-67e9-4b28-ad0c-97751e2afe10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 10:01:30.250266 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:30.250016    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 10:01:30.250266 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:30.250180    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret podName:81df3687-05a6-4b77-ac67-2f8c6626e906 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:38.25016736 +0000 UTC m=+18.091391735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret") pod "global-pull-secret-syncer-9h7fv" (UID: "81df3687-05a6-4b77-ac67-2f8c6626e906") : object "kube-system"/"original-pull-secret" not registered
Sep 08 10:01:30.451256 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:30.451207    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf4lj\" (UniqueName: \"kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj\") pod \"network-check-target-pgp7f\" (UID: \"d8cad0d3-0f69-491d-a066-2718f066d04d\") " pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:30.451419 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:30.451405    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 10:01:30.451493 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:30.451430    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 10:01:30.451493 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:30.451444    2559 projected.go:196] Error preparing data for projected volume kube-api-access-hf4lj for pod openshift-network-diagnostics/network-check-target-pgp7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 10:01:30.451591 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:30.451508    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj podName:d8cad0d3-0f69-491d-a066-2718f066d04d nodeName:}" failed. No retries permitted until 2026-09-08 10:01:38.45148978 +0000 UTC m=+18.292714160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hf4lj" (UniqueName: "kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj") pod "network-check-target-pgp7f" (UID: "d8cad0d3-0f69-491d-a066-2718f066d04d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 10:01:30.630907 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:30.630873    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:30.631054 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:30.630993    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:30.631470 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:30.631448    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:30.631827 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:30.631558    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:31.630896 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:31.630856    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:31.631066 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:31.630972    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:32.631118 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:32.631049    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:32.631603 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:32.631192    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:32.631603 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:32.631266    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:32.631603 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:32.631390    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:33.630792 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:33.630759    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:33.630960 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:33.630876    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:34.630255 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:34.630200    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:34.630673 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:34.630336    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:34.630673 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:34.630412    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:34.630673 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:34.630529    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:35.630005 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:35.629954    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:35.630107 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:35.630038    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:36.631037 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:36.631009    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:36.631456 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:36.631009    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:36.631456 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:36.631141    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:36.631456 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:36.631235    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:37.631033 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:37.630998    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:37.631597 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:37.631118    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:38.307455 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:38.307416    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs\") pod \"network-metrics-daemon-2pgcf\" (UID: \"a8a40235-67e9-4b28-ad0c-97751e2afe10\") " pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:38.307629 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:38.307463    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:38.307629 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:38.307572    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 10:01:38.307730 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:38.307631    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret podName:81df3687-05a6-4b77-ac67-2f8c6626e906 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:54.307612606 +0000 UTC m=+34.148836993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret") pod "global-pull-secret-syncer-9h7fv" (UID: "81df3687-05a6-4b77-ac67-2f8c6626e906") : object "kube-system"/"original-pull-secret" not registered
Sep 08 10:01:38.307730 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:38.307570    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 10:01:38.307730 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:38.307721    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs podName:a8a40235-67e9-4b28-ad0c-97751e2afe10 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:54.307699844 +0000 UTC m=+34.148924218 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs") pod "network-metrics-daemon-2pgcf" (UID: "a8a40235-67e9-4b28-ad0c-97751e2afe10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 10:01:38.509255 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:38.509202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf4lj\" (UniqueName: \"kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj\") pod \"network-check-target-pgp7f\" (UID: \"d8cad0d3-0f69-491d-a066-2718f066d04d\") " pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:38.509425 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:38.509378    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 10:01:38.509425 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:38.509401    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 10:01:38.509425 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:38.509411    2559 projected.go:196] Error preparing data for projected volume kube-api-access-hf4lj for pod openshift-network-diagnostics/network-check-target-pgp7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 10:01:38.509538 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:38.509464    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj podName:d8cad0d3-0f69-491d-a066-2718f066d04d nodeName:}" failed. No retries permitted until 2026-09-08 10:01:54.509450546 +0000 UTC m=+34.350674907 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hf4lj" (UniqueName: "kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj") pod "network-check-target-pgp7f" (UID: "d8cad0d3-0f69-491d-a066-2718f066d04d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 10:01:38.630748 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:38.630668    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:38.630748 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:38.630714    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:38.630963 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:38.630790    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:38.630963 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:38.630879    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:39.630629 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:39.630602    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:39.631172 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:39.630711    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:40.630931 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:40.630900    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:40.631356 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:40.631021    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:40.631443 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:40.631395    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:40.631501 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:40.631483    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:41.630635 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.630345    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:41.630749 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:41.630706    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:41.680846 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.680816    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm" event={"ID":"dee1a539-f4cb-40e2-af51-a67bc1d9dccc","Type":"ContainerStarted","Data":"4bf207d4d2745958bff825ce63aaa62b4c47a566057b834c7af6f71c3b6b93f8"}
Sep 08 10:01:41.682398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.682376    2559 generic.go:356] "Generic (PLEG): container finished" podID="6009be1324bc0532adc60e45c833edd7" containerID="8509a5839331c00a628da96139c78135d7271857bf9e725912e610e8b36f46d3" exitCode=0
Sep 08 10:01:41.682484 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.682445    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal" event={"ID":"6009be1324bc0532adc60e45c833edd7","Type":"ContainerDied","Data":"8509a5839331c00a628da96139c78135d7271857bf9e725912e610e8b36f46d3"}
Sep 08 10:01:41.684014 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.683988    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-212.ec2.internal" event={"ID":"052f4ba795232d6489bc9a1bde394dc3","Type":"ContainerStarted","Data":"f5ced44d46fd68c68b83fa5842cb9bb2b433dd4894dc4d1ed93170442b774038"}
Sep 08 10:01:41.685839 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.685814    2559 generic.go:356] "Generic (PLEG): container finished" podID="59f5177b-f1ce-4ef1-b7c2-752e58d5e27b" containerID="00d752643be8fb55a06adbac8f530d3f8aeeeb2353d94bb3df2ad468082d229f" exitCode=0
Sep 08 10:01:41.685973 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.685900    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jxq2" event={"ID":"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b","Type":"ContainerDied","Data":"00d752643be8fb55a06adbac8f530d3f8aeeeb2353d94bb3df2ad468082d229f"}
Sep 08 10:01:41.689497 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.689470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm" event={"ID":"55e4fc3c-4c4e-4f2b-958e-961b61133b3c","Type":"ContainerStarted","Data":"fc00ead2aebc567ec8e10c6d206bcd542cf42197322f8a284792186899aa5f09"}
Sep 08 10:01:41.689595 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.689541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm" event={"ID":"55e4fc3c-4c4e-4f2b-958e-961b61133b3c","Type":"ContainerStarted","Data":"e6ea5f1a8d3b3ac40da9ff81c7e1cd946ad5775297e5c25b1bff1aeb40e462c4"}
Sep 08 10:01:41.689656 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.689594    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm" event={"ID":"55e4fc3c-4c4e-4f2b-958e-961b61133b3c","Type":"ContainerStarted","Data":"092ca7e18e8aeb318dadec28eadf8504830c65e9aba73ceaffba0a5bc48e8114"}
Sep 08 10:01:41.689656 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.689641    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm" event={"ID":"55e4fc3c-4c4e-4f2b-958e-961b61133b3c","Type":"ContainerStarted","Data":"96deb18496539f46c2ccc6ee368a4086b9e2c3a082bcf2f19464f72e5309fcec"}
Sep 08 10:01:41.689760 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.689689    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm" event={"ID":"55e4fc3c-4c4e-4f2b-958e-961b61133b3c","Type":"ContainerStarted","Data":"b786b92a4a1b3fdd3776b9e1b9cf07887b625d90c4d046b873c563544f78d18b"}
Sep 08 10:01:41.689760 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.689737    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm" event={"ID":"55e4fc3c-4c4e-4f2b-958e-961b61133b3c","Type":"ContainerStarted","Data":"cf7042dd3c0b75795d4dddd55348de4877831d704c6ea142bf505b77f833d9ce"}
Sep 08 10:01:41.691066 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.691036    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fw22r" event={"ID":"a972fa38-95b8-4809-934a-bdf008f50db5","Type":"ContainerStarted","Data":"2d489d754159c929eba730eceabe1c32f2c5cd183bbcc0395ab1f851f94da09a"}
Sep 08 10:01:41.692653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.692632    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5lzjn" event={"ID":"ff849db0-791c-405b-bbfc-868f1b7288d2","Type":"ContainerStarted","Data":"d28fb047489e942f860904c6d66fc8e107bc35c6be82df39f3c9c0b28f66e771"}
Sep 08 10:01:41.694101 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.694082    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hp9jt" event={"ID":"2719b26a-ad31-4018-a46c-927706ebc26e","Type":"ContainerStarted","Data":"b89bf2d06776fd6b0ad59fea9307955e05a9ef792e5a79223337513a5826ca1b"}
Sep 08 10:01:41.695615 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.695592    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8sj86" event={"ID":"fdf41754-0a6d-4c9c-93c5-240108b5c9db","Type":"ContainerStarted","Data":"fa87203ed44bf12ac982c4f1471af01cfff2661f8f9a304ee73beba632f99d23"}
Sep 08 10:01:41.697046 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.697025    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w8khh" event={"ID":"62ec76ef-d924-4dbb-b66b-735e295ddc0c","Type":"ContainerStarted","Data":"b384ed326163503a62b377460adca4ba61dffa916c5c09cf56a94fe0c5e38be9"}
Sep 08 10:01:41.718533 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.718109    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fw22r" podStartSLOduration=7.096784035 podStartE2EDuration="20.718091163s" totalImagesPullingTime="13.621307128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:41.717879719 +0000 UTC m=+21.559104102" watchObservedRunningTime="2026-09-08 10:01:41.718091163 +0000 UTC m=+21.559315550"
Sep 08 10:01:41.729963 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.729917    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-212.ec2.internal" podStartSLOduration=19.729899972 podStartE2EDuration="19.729899972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:41.729822336 +0000 UTC m=+21.571046741" watchObservedRunningTime="2026-09-08 10:01:41.729899972 +0000 UTC m=+21.571124356"
Sep 08 10:01:41.755521 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.755487    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-w8khh" podStartSLOduration=3.572617236 podStartE2EDuration="21.755474463s" totalImagesPullingTime="18.182857227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:41.75513988 +0000 UTC m=+21.596364257" watchObservedRunningTime="2026-09-08 10:01:41.755474463 +0000 UTC m=+21.596698846"
Sep 08 10:01:41.776022 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.775988    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8sj86" podStartSLOduration=6.850056432 podStartE2EDuration="21.775975471s" totalImagesPullingTime="14.925919039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:41.77585745 +0000 UTC m=+21.617081838" watchObservedRunningTime="2026-09-08 10:01:41.775975471 +0000 UTC m=+21.617199854"
Sep 08 10:01:41.815573 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:41.815529    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hp9jt" podStartSLOduration=9.362921089 podStartE2EDuration="21.815516662s" totalImagesPullingTime="12.452595573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:41.815494299 +0000 UTC m=+21.656718702" watchObservedRunningTime="2026-09-08 10:01:41.815516662 +0000 UTC m=+21.656741047"
Sep 08 10:01:42.630396 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:42.630357    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:42.630551 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:42.630506    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:42.630623 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:42.630565    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:42.630721 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:42.630691    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:42.699770 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:42.699744    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gbj76" event={"ID":"27334e80-03d1-4b30-a744-8b6ea5eaf69f","Type":"ContainerStarted","Data":"915cf43fa2c1e337644c3b956504560f51477db34264cb891b0fd507b76bb017"}
Sep 08 10:01:42.701475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:42.701446    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal" event={"ID":"6009be1324bc0532adc60e45c833edd7","Type":"ContainerStarted","Data":"51eef38fc534443028372a28143553bd9aab6ecce48d1dbff9869efda7998bc6"}
Sep 08 10:01:42.714358 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:42.714289    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5lzjn" podStartSLOduration=4.5430042870000005 podStartE2EDuration="22.714276788s" totalImagesPullingTime="18.171272501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:41.831961055 +0000 UTC m=+21.673185458" watchObservedRunningTime="2026-09-08 10:01:42.714276788 +0000 UTC m=+22.555501172"
Sep 08 10:01:42.714698 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:42.714677    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gbj76" podStartSLOduration=9.087026029 podStartE2EDuration="22.714672674s" totalImagesPullingTime="13.627646645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:42.714197835 +0000 UTC m=+22.555422217" watchObservedRunningTime="2026-09-08 10:01:42.714672674 +0000 UTC m=+22.555897058"
Sep 08 10:01:42.731825 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:42.731792    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-212.ec2.internal" podStartSLOduration=20.731780811 podStartE2EDuration="20.731780811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:42.731356351 +0000 UTC m=+22.572580755" watchObservedRunningTime="2026-09-08 10:01:42.731780811 +0000 UTC m=+22.573005194"
Sep 08 10:01:43.270278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:43.270234    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 10:01:43.628958 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:43.628783    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T10:01:43.270273874Z","UUID":"276021dd-1684-479f-95ea-a37cdb736804","Handler":null,"Name":"","Endpoint":""}
Sep 08 10:01:43.630081 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:43.630058    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:43.630259 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:43.630221    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:43.630409 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:43.630393    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 10:01:43.630494 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:43.630424    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 10:01:43.705722 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:43.705700    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm" event={"ID":"dee1a539-f4cb-40e2-af51-a67bc1d9dccc","Type":"ContainerStarted","Data":"74097ce6bef8da54e057abf3cf019c747fcf77643faf2110e791471f93f295f2"}
Sep 08 10:01:44.630320 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:44.630118    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:44.630483 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:44.630144    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:44.630483 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:44.630447    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:44.630483 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:44.630460    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:44.712426 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:44.712350    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm" event={"ID":"55e4fc3c-4c4e-4f2b-958e-961b61133b3c","Type":"ContainerStarted","Data":"f95b45c79885975ad86229e2d5ac928913241943db53853ea143734745152260"}
Sep 08 10:01:44.714797 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:44.714774    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm" event={"ID":"dee1a539-f4cb-40e2-af51-a67bc1d9dccc","Type":"ContainerStarted","Data":"fda484f08412c4fff51e48a4613a6335b97e95d1e758c9d18ae7a22681dd94c7"}
Sep 08 10:01:44.731513 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:44.731472    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wjgm" podStartSLOduration=6.987705903 podStartE2EDuration="24.731455058s" totalImagesPullingTime="17.743749155s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:44.73123858 +0000 UTC m=+24.572462963" watchObservedRunningTime="2026-09-08 10:01:44.731455058 +0000 UTC m=+24.572679441"
Sep 08 10:01:45.630416 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:45.630386    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:45.630600 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:45.630506    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:45.643199 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:45.643173    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-w8khh"
Sep 08 10:01:45.643835 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:45.643809    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-w8khh"
Sep 08 10:01:45.717181 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:45.717131    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-w8khh"
Sep 08 10:01:45.717588 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:45.717560    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w8khh"
Sep 08 10:01:46.631125 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:46.631058    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:46.631270 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:46.631066    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:46.631270 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:46.631157    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:46.631270 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:46.631230    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:46.720397 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:46.720361    2559 generic.go:356] "Generic (PLEG): container finished" podID="59f5177b-f1ce-4ef1-b7c2-752e58d5e27b" containerID="cdad86245b9e34d42b233159ba7629cc48d7b0d302dd3cb6347dcad8d2dc684a" exitCode=0
Sep 08 10:01:46.720882 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:46.720444    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jxq2" event={"ID":"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b","Type":"ContainerDied","Data":"cdad86245b9e34d42b233159ba7629cc48d7b0d302dd3cb6347dcad8d2dc684a"}
Sep 08 10:01:46.726551 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:46.726519    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm" event={"ID":"55e4fc3c-4c4e-4f2b-958e-961b61133b3c","Type":"ContainerStarted","Data":"045c760f6058d119347d3c302aa869d07e0655f245177aad40ffd0e618167a82"}
Sep 08 10:01:46.726766 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:46.726745    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:46.764232 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:46.764185    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm" podStartSLOduration=7.528204078 podStartE2EDuration="25.764172205s" totalImagesPullingTime="18.235968127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:01:46.763951239 +0000 UTC m=+26.605175622" watchObservedRunningTime="2026-09-08 10:01:46.764172205 +0000 UTC m=+26.605396588"
Sep 08 10:01:47.631069 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:47.630901    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:47.631202 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:47.631140    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:47.731631 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:47.731556    2559 generic.go:356] "Generic (PLEG): container finished" podID="59f5177b-f1ce-4ef1-b7c2-752e58d5e27b" containerID="5b1027467ca5d26367e54c09ea6fa4c151acd5925339fafd238f7d108511285d" exitCode=0
Sep 08 10:01:47.732055 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:47.731641    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jxq2" event={"ID":"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b","Type":"ContainerDied","Data":"5b1027467ca5d26367e54c09ea6fa4c151acd5925339fafd238f7d108511285d"}
Sep 08 10:01:47.732193 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:47.732172    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:47.732473 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:47.732457    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:47.747980 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:47.747959    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:47.748316 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:47.748299    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:01:48.201582 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:48.201502    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9h7fv"]
Sep 08 10:01:48.201733 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:48.201651    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:48.201790 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:48.201749    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:48.204917 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:48.204874    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2pgcf"]
Sep 08 10:01:48.205081 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:48.205058    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:48.205257 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:48.205203    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:48.205817 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:48.205794    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pgp7f"]
Sep 08 10:01:48.205966 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:48.205949    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:48.206061 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:48.206041    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:48.735894 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:48.735813    2559 generic.go:356] "Generic (PLEG): container finished" podID="59f5177b-f1ce-4ef1-b7c2-752e58d5e27b" containerID="333feeb4d722c447797f2195169550388736d1ba3b668ec0a9048a24b3c4416d" exitCode=0
Sep 08 10:01:48.736426 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:48.735891    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jxq2" event={"ID":"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b","Type":"ContainerDied","Data":"333feeb4d722c447797f2195169550388736d1ba3b668ec0a9048a24b3c4416d"}
Sep 08 10:01:49.630184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:49.630111    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:49.630327 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:49.630218    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:49.630327 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:49.630269    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:49.630405 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:49.630350    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:50.633291 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:50.632799    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:50.633291 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:50.632919    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:51.630456 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:51.630429    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:51.630456 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:51.630445    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:51.630647 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:51.630522    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:51.630705 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:51.630680    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:52.630226 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:52.630195    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:52.630649 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:52.630338    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgp7f" podUID="d8cad0d3-0f69-491d-a066-2718f066d04d"
Sep 08 10:01:53.630409 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:53.630204    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:53.630799 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:53.630216    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:53.630799 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:53.630524    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9h7fv" podUID="81df3687-05a6-4b77-ac67-2f8c6626e906"
Sep 08 10:01:53.630799 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:53.630581    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2pgcf" podUID="a8a40235-67e9-4b28-ad0c-97751e2afe10"
Sep 08 10:01:53.963640 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:53.963610    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-212.ec2.internal" event="NodeReady"
Sep 08 10:01:53.963808 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:53.963747    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 10:01:53.997662 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:53.997635    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d946c986b-wvnk5"]
Sep 08 10:01:54.028216 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.028183    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q"]
Sep 08 10:01:54.028590 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.028561    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.032321 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.032286    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 10:01:54.033349 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.033299    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 10:01:54.034446 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.034428    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9wz2f\""
Sep 08 10:01:54.034640 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.034623    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 10:01:54.039336 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.039308    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 10:01:54.042185 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.042168    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5fbcc7c578-fzqlh"]
Sep 08 10:01:54.042323 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.042308    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q"
Sep 08 10:01:54.045818 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.045758    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 08 10:01:54.045944 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.045834    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 08 10:01:54.046212 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.046191    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5xcg4\""
Sep 08 10:01:54.046377 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.046361    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 08 10:01:54.046483 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.046466    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 08 10:01:54.061596 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.061568    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-7cwbx"]
Sep 08 10:01:54.061702 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.061661    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.064670 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.064506    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 08 10:01:54.064670 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.064520    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 08 10:01:54.064821 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.064792    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 10:01:54.064944 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.064930    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 08 10:01:54.065359 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.065321    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cdwlb\""
Sep 08 10:01:54.065665 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.065649    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 08 10:01:54.065840 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.065827    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 10:01:54.079939 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.079882    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-pm7xz"]
Sep 08 10:01:54.080033 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.079961    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx"
Sep 08 10:01:54.082461 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.082382    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jjps9\""
Sep 08 10:01:54.082461 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.082372    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 08 10:01:54.082678 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.082659    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 08 10:01:54.092577 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.092558    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-hpbwn"]
Sep 08 10:01:54.092709 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.092681    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:01:54.094785 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.094767    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cq7p6\""
Sep 08 10:01:54.095151 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.095134    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 08 10:01:54.095656 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.095624    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 08 10:01:54.095769 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.095709    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 08 10:01:54.096520 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.096501    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 08 10:01:54.100773 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.100756    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 08 10:01:54.109283 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.109266    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c"]
Sep 08 10:01:54.109363 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.109317    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-hpbwn"
Sep 08 10:01:54.112239 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.112169    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 08 10:01:54.112239 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.112169    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xgjs7\""
Sep 08 10:01:54.112478 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.112458    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 08 10:01:54.120721 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.120696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwrwt\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-kube-api-access-zwrwt\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.120809 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.120749    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7zf5\" (UniqueName: \"kubernetes.io/projected/5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9-kube-api-access-m7zf5\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ls69q\" (UID: \"5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q"
Sep 08 10:01:54.120809 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.120785    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7149cdff-fd81-49d1-9df7-9757dfd544d3-trusted-ca\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.120909 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.120829    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-bound-sa-token\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.120975 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.120926    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7149cdff-fd81-49d1-9df7-9757dfd544d3-image-registry-private-configuration\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.121030 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.121009    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7149cdff-fd81-49d1-9df7-9757dfd544d3-ca-trust-extracted\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.121085 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.121036    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7149cdff-fd81-49d1-9df7-9757dfd544d3-installation-pull-secrets\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.121085 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.121066    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ls69q\" (UID: \"5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q"
Sep 08 10:01:54.121186 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.121136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ls69q\" (UID: \"5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q"
Sep 08 10:01:54.121186 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.121179    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.121315 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.121202    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-certificates\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.121690 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.121674    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"]
Sep 08 10:01:54.121837 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.121822    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c"
Sep 08 10:01:54.123956 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.123938    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 08 10:01:54.124045 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.124034    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 08 10:01:54.124362 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.124342    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 08 10:01:54.124455 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.124372    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 08 10:01:54.124677 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.124660    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jvxqt\""
Sep 08 10:01:54.141541 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.141524    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-hpbwn"]
Sep 08 10:01:54.141633 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.141554    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c"]
Sep 08 10:01:54.141633 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.141572    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d946c986b-wvnk5"]
Sep 08 10:01:54.141633 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.141594    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q"]
Sep 08 10:01:54.141633 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.141612    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5fbcc7c578-fzqlh"]
Sep 08 10:01:54.141633 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.141629    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"]
Sep 08 10:01:54.141813 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.141658    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"]
Sep 08 10:01:54.141813 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.141721    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"
Sep 08 10:01:54.144083 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.144063    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 08 10:01:54.144227 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.144201    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c7kg4\""
Sep 08 10:01:54.144348 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.144276    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 08 10:01:54.144406 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.144352    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 08 10:01:54.152287 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.152271    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-s5rlp"]
Sep 08 10:01:54.152406 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.152393    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:01:54.155076 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.155037    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 08 10:01:54.155378 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.155357    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2pfvl\""
Sep 08 10:01:54.155746 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.155727    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 10:01:54.155891 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.155873    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 10:01:54.156581 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.156564    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 08 10:01:54.172859 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.172842    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fnmlf"]
Sep 08 10:01:54.172986 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.172969    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-s5rlp"
Sep 08 10:01:54.175434 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.175402    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 10:01:54.175530 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.175430    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 10:01:54.175725 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.175662    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jg6z6\""
Sep 08 10:01:54.185998 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.185981    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5h7jm"]
Sep 08 10:01:54.186113 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.186095    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:54.188902 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.188878    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pd9dz\""
Sep 08 10:01:54.189183 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.189167    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 10:01:54.189449 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.189434    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 10:01:54.205628 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.205607    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-dwmlf"]
Sep 08 10:01:54.205744 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.205724    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5h7jm"
Sep 08 10:01:54.208164 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.208147    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 10:01:54.208261 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.208167    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 10:01:54.208261 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.208218    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-94npr\""
Sep 08 10:01:54.208363 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.208302    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 10:01:54.221482 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221429    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.221482 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221461    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgflf\" (UniqueName: \"kubernetes.io/projected/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-kube-api-access-pgflf\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.221604 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221493    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13d91b33-2bf0-4bf4-b866-8b6bc251c280-nginx-conf\") pod \"networking-console-plugin-79977f9844-7cwbx\" (UID: \"13d91b33-2bf0-4bf4-b866-8b6bc251c280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx"
Sep 08 10:01:54.221604 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221533    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmvwf\" (UniqueName: \"kubernetes.io/projected/633deb4b-3d4a-47c5-b6ef-a8d8a803c521-kube-api-access-dmvwf\") pod \"service-ca-operator-859f94c4b-p9j4c\" (UID: \"633deb4b-3d4a-47c5-b6ef-a8d8a803c521\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c"
Sep 08 10:01:54.221711 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221639    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7149cdff-fd81-49d1-9df7-9757dfd544d3-ca-trust-extracted\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.221711 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221667    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7149cdff-fd81-49d1-9df7-9757dfd544d3-installation-pull-secrets\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.221711 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ls69q\" (UID: \"5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q"
Sep 08 10:01:54.221836 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221719    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/633deb4b-3d4a-47c5-b6ef-a8d8a803c521-serving-cert\") pod \"service-ca-operator-859f94c4b-p9j4c\" (UID: \"633deb4b-3d4a-47c5-b6ef-a8d8a803c521\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c"
Sep 08 10:01:54.221836 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lzm7\" (UniqueName: \"kubernetes.io/projected/4e917bce-6da7-4d7a-bbb2-bd09d8086599-kube-api-access-8lzm7\") pod \"console-operator-85b454874b-pm7xz\" (UID: \"4e917bce-6da7-4d7a-bbb2-bd09d8086599\") " pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:01:54.221836 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221768    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-5p6mp\" (UID: \"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"
Sep 08 10:01:54.221836 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9bws\" (UniqueName: \"kubernetes.io/projected/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-kube-api-access-d9bws\") pod \"cluster-samples-operator-7cdcfcf76c-5p6mp\" (UID: \"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"
Sep 08 10:01:54.222018 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221896    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7cwbx\" (UID: \"13d91b33-2bf0-4bf4-b866-8b6bc251c280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx"
Sep 08 10:01:54.222018 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ls69q\" (UID: \"5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q"
Sep 08 10:01:54.222018 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e917bce-6da7-4d7a-bbb2-bd09d8086599-config\") pod \"console-operator-85b454874b-pm7xz\" (UID: \"4e917bce-6da7-4d7a-bbb2-bd09d8086599\") " pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:01:54.222018 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e917bce-6da7-4d7a-bbb2-bd09d8086599-serving-cert\") pod \"console-operator-85b454874b-pm7xz\" (UID: \"4e917bce-6da7-4d7a-bbb2-bd09d8086599\") " pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:01:54.222018 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.221986    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7149cdff-fd81-49d1-9df7-9757dfd544d3-ca-trust-extracted\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222019    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222046    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-certificates\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-default-certificate\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.222091    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.222106    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d946c986b-wvnk5: secret "image-registry-tls" not found
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222142    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwrwt\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-kube-api-access-zwrwt\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.222167    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls podName:7149cdff-fd81-49d1-9df7-9757dfd544d3 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:54.722148115 +0000 UTC m=+34.563372484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls") pod "image-registry-d946c986b-wvnk5" (UID: "7149cdff-fd81-49d1-9df7-9757dfd544d3") : secret "image-registry-tls" not found
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222206    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7zf5\" (UniqueName: \"kubernetes.io/projected/5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9-kube-api-access-m7zf5\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ls69q\" (UID: \"5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q"
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7149cdff-fd81-49d1-9df7-9757dfd544d3-trusted-ca\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-bound-sa-token\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222287    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-7cwbx"]
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222294    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ls69q\" (UID: \"5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q"
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222312    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7149cdff-fd81-49d1-9df7-9757dfd544d3-image-registry-private-configuration\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222323    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-pm7xz"]
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222341    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-stats-auth\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.222398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222368    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v9wz\" (UniqueName: \"kubernetes.io/projected/15e7a59c-e003-45d4-8878-63728f83b9dc-kube-api-access-8v9wz\") pod \"volume-data-source-validator-695c895bd6-hpbwn\" (UID: \"15e7a59c-e003-45d4-8878-63728f83b9dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-hpbwn"
Sep 08 10:01:54.223177 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222397    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/633deb4b-3d4a-47c5-b6ef-a8d8a803c521-config\") pod \"service-ca-operator-859f94c4b-p9j4c\" (UID: \"633deb4b-3d4a-47c5-b6ef-a8d8a803c521\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c"
Sep 08 10:01:54.223177 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222408    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.223177 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222427    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e917bce-6da7-4d7a-bbb2-bd09d8086599-trusted-ca\") pod \"console-operator-85b454874b-pm7xz\" (UID: \"4e917bce-6da7-4d7a-bbb2-bd09d8086599\") " pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:01:54.223177 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222347    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-s5rlp"]
Sep 08 10:01:54.223177 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222531    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"]
Sep 08 10:01:54.223177 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222552    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5h7jm"]
Sep 08 10:01:54.223177 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222571    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fnmlf"]
Sep 08 10:01:54.223177 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222594    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-dwmlf"]
Sep 08 10:01:54.223177 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.222699    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-certificates\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.223559 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.223343    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7149cdff-fd81-49d1-9df7-9757dfd544d3-trusted-ca\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.224592 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.224572    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 10:01:54.225067 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.225047    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 08 10:01:54.225278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.225257    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 08 10:01:54.225675 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.225506    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 10:01:54.225675 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.225590    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5pkmn\""
Sep 08 10:01:54.226428 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.226360    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ls69q\" (UID: \"5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q"
Sep 08 10:01:54.226428 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.226419    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7149cdff-fd81-49d1-9df7-9757dfd544d3-image-registry-private-configuration\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.226561 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.226454    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7149cdff-fd81-49d1-9df7-9757dfd544d3-installation-pull-secrets\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.230947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.230919    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 08 10:01:54.231556 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.231538    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwrwt\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-kube-api-access-zwrwt\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.231752 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.231734    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7zf5\" (UniqueName: \"kubernetes.io/projected/5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9-kube-api-access-m7zf5\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ls69q\" (UID: \"5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q"
Sep 08 10:01:54.231839 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.231816    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-bound-sa-token\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.323772 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.323741    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/633deb4b-3d4a-47c5-b6ef-a8d8a803c521-config\") pod \"service-ca-operator-859f94c4b-p9j4c\" (UID: \"633deb4b-3d4a-47c5-b6ef-a8d8a803c521\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c"
Sep 08 10:01:54.323910 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.323786    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e917bce-6da7-4d7a-bbb2-bd09d8086599-trusted-ca\") pod \"console-operator-85b454874b-pm7xz\" (UID: \"4e917bce-6da7-4d7a-bbb2-bd09d8086599\") " pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:01:54.323910 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.323817    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lzm7\" (UniqueName: \"kubernetes.io/projected/4e917bce-6da7-4d7a-bbb2-bd09d8086599-kube-api-access-8lzm7\") pod \"console-operator-85b454874b-pm7xz\" (UID: \"4e917bce-6da7-4d7a-bbb2-bd09d8086599\") " pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:01:54.323910 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.323851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:54.323910 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.323877    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/633deb4b-3d4a-47c5-b6ef-a8d8a803c521-serving-cert\") pod \"service-ca-operator-859f94c4b-p9j4c\" (UID: \"633deb4b-3d4a-47c5-b6ef-a8d8a803c521\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c"
Sep 08 10:01:54.323910 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.323899    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bd18742f-fe5e-4fcb-b416-b566be8344e4-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:01:54.324156 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.323977    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-snapshots\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.324156 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324022    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7cwbx\" (UID: \"13d91b33-2bf0-4bf4-b866-8b6bc251c280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx"
Sep 08 10:01:54.324156 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324052    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.324156 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:01:54.324156 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324110    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cacaccdd-dd0d-4bcd-a974-a065758240f7-config-volume\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:54.324156 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-tmp\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.324467 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324210    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-default-certificate\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.324467 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.324303    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 10:01:54.324467 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.324370    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert podName:13d91b33-2bf0-4bf4-b866-8b6bc251c280 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:54.824355642 +0000 UTC m=+34.665580006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-7cwbx" (UID: "13d91b33-2bf0-4bf4-b866-8b6bc251c280") : secret "networking-console-plugin-cert" not found
Sep 08 10:01:54.324467 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.324467 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324455    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v22n9\" (UniqueName: \"kubernetes.io/projected/2461e11a-d91a-4e42-bb9e-8646646fdf40-kube-api-access-v22n9\") pod \"ingress-canary-5h7jm\" (UID: \"2461e11a-d91a-4e42-bb9e-8646646fdf40\") " pod="openshift-ingress-canary/ingress-canary-5h7jm"
Sep 08 10:01:54.324713 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324488    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2461e11a-d91a-4e42-bb9e-8646646fdf40-cert\") pod \"ingress-canary-5h7jm\" (UID: \"2461e11a-d91a-4e42-bb9e-8646646fdf40\") " pod="openshift-ingress-canary/ingress-canary-5h7jm"
Sep 08 10:01:54.324713 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324509    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cacaccdd-dd0d-4bcd-a974-a065758240f7-tmp-dir\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:54.324713 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324526    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/633deb4b-3d4a-47c5-b6ef-a8d8a803c521-config\") pod \"service-ca-operator-859f94c4b-p9j4c\" (UID: \"633deb4b-3d4a-47c5-b6ef-a8d8a803c521\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c"
Sep 08 10:01:54.324713 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-5p6mp\" (UID: \"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"
Sep 08 10:01:54.324713 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v9wz\" (UniqueName: \"kubernetes.io/projected/15e7a59c-e003-45d4-8878-63728f83b9dc-kube-api-access-8v9wz\") pod \"volume-data-source-validator-695c895bd6-hpbwn\" (UID: \"15e7a59c-e003-45d4-8878-63728f83b9dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-hpbwn"
Sep 08 10:01:54.324713 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.324622    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 10:01:54.324713 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324635    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.324713 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324660    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgflf\" (UniqueName: \"kubernetes.io/projected/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-kube-api-access-pgflf\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.324713 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.324685    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs podName:132efab2-1cc6-4fac-a9ef-61d9d8ab4c48 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:54.824669126 +0000 UTC m=+34.665893502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs") pod "router-default-5fbcc7c578-fzqlh" (UID: "132efab2-1cc6-4fac-a9ef-61d9d8ab4c48") : secret "router-metrics-certs-default" not found
Sep 08 10:01:54.324713 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.324716    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.324759    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls podName:e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:54.824745658 +0000 UTC m=+34.665970030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-5p6mp" (UID: "e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464") : secret "samples-operator-tls" not found
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13d91b33-2bf0-4bf4-b866-8b6bc251c280-nginx-conf\") pod \"networking-console-plugin-79977f9844-7cwbx\" (UID: \"13d91b33-2bf0-4bf4-b866-8b6bc251c280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx"
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.324783    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle podName:132efab2-1cc6-4fac-a9ef-61d9d8ab4c48 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:54.82476646 +0000 UTC m=+34.665990834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle") pod "router-default-5fbcc7c578-fzqlh" (UID: "132efab2-1cc6-4fac-a9ef-61d9d8ab4c48") : configmap references non-existent config key: service-ca.crt
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324781    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e917bce-6da7-4d7a-bbb2-bd09d8086599-trusted-ca\") pod \"console-operator-85b454874b-pm7xz\" (UID: \"4e917bce-6da7-4d7a-bbb2-bd09d8086599\") " pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324821    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmvwf\" (UniqueName: \"kubernetes.io/projected/633deb4b-3d4a-47c5-b6ef-a8d8a803c521-kube-api-access-dmvwf\") pod \"service-ca-operator-859f94c4b-p9j4c\" (UID: \"633deb4b-3d4a-47c5-b6ef-a8d8a803c521\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c"
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324886    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-serving-cert\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324915    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9bws\" (UniqueName: \"kubernetes.io/projected/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-kube-api-access-d9bws\") pod \"cluster-samples-operator-7cdcfcf76c-5p6mp\" (UID: \"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324943    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs\") pod \"network-metrics-daemon-2pgcf\" (UID: \"a8a40235-67e9-4b28-ad0c-97751e2afe10\") " pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e917bce-6da7-4d7a-bbb2-bd09d8086599-config\") pod \"console-operator-85b454874b-pm7xz\" (UID: \"4e917bce-6da7-4d7a-bbb2-bd09d8086599\") " pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.324996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e917bce-6da7-4d7a-bbb2-bd09d8086599-serving-cert\") pod \"console-operator-85b454874b-pm7xz\" (UID: \"4e917bce-6da7-4d7a-bbb2-bd09d8086599\") " pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.325028    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfzfn\" (UniqueName: \"kubernetes.io/projected/bd18742f-fe5e-4fcb-b416-b566be8344e4-kube-api-access-zfzfn\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.325059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j2t7\" (UniqueName: \"kubernetes.io/projected/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-kube-api-access-4j2t7\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.325065    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 10:01:54.325184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.325085    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvw8k\" (UniqueName: \"kubernetes.io/projected/65c33bab-504d-48c2-96f0-5ba84ff76f8f-kube-api-access-nvw8k\") pod \"network-check-source-685456655f-s5rlp\" (UID: \"65c33bab-504d-48c2-96f0-5ba84ff76f8f\") " pod="openshift-network-diagnostics/network-check-source-685456655f-s5rlp"
Sep 08 10:01:54.325917 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.325118    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cthk\" (UniqueName: \"kubernetes.io/projected/cacaccdd-dd0d-4bcd-a974-a065758240f7-kube-api-access-8cthk\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:54.325917 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.325144    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs podName:a8a40235-67e9-4b28-ad0c-97751e2afe10 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:26.325126466 +0000 UTC m=+66.166350828 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs") pod "network-metrics-daemon-2pgcf" (UID: "a8a40235-67e9-4b28-ad0c-97751e2afe10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 10:01:54.325917 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.325226    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:54.325917 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.325288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-stats-auth\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.325917 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.325329    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 10:01:54.325917 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.325383    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret podName:81df3687-05a6-4b77-ac67-2f8c6626e906 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:26.325369311 +0000 UTC m=+66.166593697 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret") pod "global-pull-secret-syncer-9h7fv" (UID: "81df3687-05a6-4b77-ac67-2f8c6626e906") : object "kube-system"/"original-pull-secret" not registered
Sep 08 10:01:54.325917 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.325749    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e917bce-6da7-4d7a-bbb2-bd09d8086599-config\") pod \"console-operator-85b454874b-pm7xz\" (UID: \"4e917bce-6da7-4d7a-bbb2-bd09d8086599\") " pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:01:54.325917 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.325768    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13d91b33-2bf0-4bf4-b866-8b6bc251c280-nginx-conf\") pod \"networking-console-plugin-79977f9844-7cwbx\" (UID: \"13d91b33-2bf0-4bf4-b866-8b6bc251c280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx"
Sep 08 10:01:54.326990 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.326966    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/633deb4b-3d4a-47c5-b6ef-a8d8a803c521-serving-cert\") pod \"service-ca-operator-859f94c4b-p9j4c\" (UID: \"633deb4b-3d4a-47c5-b6ef-a8d8a803c521\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c"
Sep 08 10:01:54.327111 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.327090    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-default-certificate\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.327839 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.327822    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e917bce-6da7-4d7a-bbb2-bd09d8086599-serving-cert\") pod \"console-operator-85b454874b-pm7xz\" (UID: \"4e917bce-6da7-4d7a-bbb2-bd09d8086599\") " pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:01:54.328078 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.328058    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-stats-auth\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.332851 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.332832    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v9wz\" (UniqueName: \"kubernetes.io/projected/15e7a59c-e003-45d4-8878-63728f83b9dc-kube-api-access-8v9wz\") pod \"volume-data-source-validator-695c895bd6-hpbwn\" (UID: \"15e7a59c-e003-45d4-8878-63728f83b9dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-hpbwn"
Sep 08 10:01:54.332944 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.332836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmvwf\" (UniqueName: \"kubernetes.io/projected/633deb4b-3d4a-47c5-b6ef-a8d8a803c521-kube-api-access-dmvwf\") pod \"service-ca-operator-859f94c4b-p9j4c\" (UID: \"633deb4b-3d4a-47c5-b6ef-a8d8a803c521\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c"
Sep 08 10:01:54.333365 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.333344    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgflf\" (UniqueName: \"kubernetes.io/projected/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-kube-api-access-pgflf\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.333556 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.333535    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lzm7\" (UniqueName: \"kubernetes.io/projected/4e917bce-6da7-4d7a-bbb2-bd09d8086599-kube-api-access-8lzm7\") pod \"console-operator-85b454874b-pm7xz\" (UID: \"4e917bce-6da7-4d7a-bbb2-bd09d8086599\") " pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:01:54.334674 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.334632    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9bws\" (UniqueName: \"kubernetes.io/projected/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-kube-api-access-d9bws\") pod \"cluster-samples-operator-7cdcfcf76c-5p6mp\" (UID: \"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"
Sep 08 10:01:54.352184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.352159    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q"
Sep 08 10:01:54.402908 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.402884    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:01:54.417964 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.417933    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-hpbwn"
Sep 08 10:01:54.425791 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.425764    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:01:54.425878 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.425801    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cacaccdd-dd0d-4bcd-a974-a065758240f7-config-volume\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:54.425878 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.425820    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-tmp\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.425969 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.425881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v22n9\" (UniqueName: \"kubernetes.io/projected/2461e11a-d91a-4e42-bb9e-8646646fdf40-kube-api-access-v22n9\") pod \"ingress-canary-5h7jm\" (UID: \"2461e11a-d91a-4e42-bb9e-8646646fdf40\") " pod="openshift-ingress-canary/ingress-canary-5h7jm"
Sep 08 10:01:54.425969 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.425912    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2461e11a-d91a-4e42-bb9e-8646646fdf40-cert\") pod \"ingress-canary-5h7jm\" (UID: \"2461e11a-d91a-4e42-bb9e-8646646fdf40\") " pod="openshift-ingress-canary/ingress-canary-5h7jm"
Sep 08 10:01:54.425969 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.425926    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 10:01:54.425969 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.425937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cacaccdd-dd0d-4bcd-a974-a065758240f7-tmp-dir\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:54.426152 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.425988    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls podName:bd18742f-fe5e-4fcb-b416-b566be8344e4 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:54.925968265 +0000 UTC m=+34.767192630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wgqz8" (UID: "bd18742f-fe5e-4fcb-b416-b566be8344e4") : secret "cluster-monitoring-operator-tls" not found
Sep 08 10:01:54.426152 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.426051    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.426152 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.426084    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-serving-cert\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.426152 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.426132    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzfn\" (UniqueName: \"kubernetes.io/projected/bd18742f-fe5e-4fcb-b416-b566be8344e4-kube-api-access-zfzfn\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:01:54.426382 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.426166    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j2t7\" (UniqueName: \"kubernetes.io/projected/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-kube-api-access-4j2t7\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.426382 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.426182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-tmp\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.426382 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.426272    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cacaccdd-dd0d-4bcd-a974-a065758240f7-tmp-dir\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:54.426382 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.426187    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvw8k\" (UniqueName: \"kubernetes.io/projected/65c33bab-504d-48c2-96f0-5ba84ff76f8f-kube-api-access-nvw8k\") pod \"network-check-source-685456655f-s5rlp\" (UID: \"65c33bab-504d-48c2-96f0-5ba84ff76f8f\") " pod="openshift-network-diagnostics/network-check-source-685456655f-s5rlp"
Sep 08 10:01:54.426382 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.426319    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cthk\" (UniqueName: \"kubernetes.io/projected/cacaccdd-dd0d-4bcd-a974-a065758240f7-kube-api-access-8cthk\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:54.426610 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.426408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:54.426610 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.426438    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bd18742f-fe5e-4fcb-b416-b566be8344e4-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:01:54.426610 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.426490    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cacaccdd-dd0d-4bcd-a974-a065758240f7-config-volume\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:54.426610 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.426504    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 10:01:54.426610 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.426563    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls podName:cacaccdd-dd0d-4bcd-a974-a065758240f7 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:54.926539801 +0000 UTC m=+34.767764173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls") pod "dns-default-fnmlf" (UID: "cacaccdd-dd0d-4bcd-a974-a065758240f7") : secret "dns-default-metrics-tls" not found
Sep 08 10:01:54.426610 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.426593    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-snapshots\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.426922 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.426639    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.427220 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.427176    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.427220 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.427189    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bd18742f-fe5e-4fcb-b416-b566be8344e4-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:01:54.427401 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.427236    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-snapshots\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.427401 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.427305    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.428872 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.428839    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2461e11a-d91a-4e42-bb9e-8646646fdf40-cert\") pod \"ingress-canary-5h7jm\" (UID: \"2461e11a-d91a-4e42-bb9e-8646646fdf40\") " pod="openshift-ingress-canary/ingress-canary-5h7jm"
Sep 08 10:01:54.429009 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.428991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-serving-cert\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.430912 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.430893    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c"
Sep 08 10:01:54.435209 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.435185    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvw8k\" (UniqueName: \"kubernetes.io/projected/65c33bab-504d-48c2-96f0-5ba84ff76f8f-kube-api-access-nvw8k\") pod \"network-check-source-685456655f-s5rlp\" (UID: \"65c33bab-504d-48c2-96f0-5ba84ff76f8f\") " pod="openshift-network-diagnostics/network-check-source-685456655f-s5rlp"
Sep 08 10:01:54.435402 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.435372    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v22n9\" (UniqueName: \"kubernetes.io/projected/2461e11a-d91a-4e42-bb9e-8646646fdf40-kube-api-access-v22n9\") pod \"ingress-canary-5h7jm\" (UID: \"2461e11a-d91a-4e42-bb9e-8646646fdf40\") " pod="openshift-ingress-canary/ingress-canary-5h7jm"
Sep 08 10:01:54.435519 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.435498    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cthk\" (UniqueName: \"kubernetes.io/projected/cacaccdd-dd0d-4bcd-a974-a065758240f7-kube-api-access-8cthk\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:54.436088 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.436063    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j2t7\" (UniqueName: \"kubernetes.io/projected/1ba9dc85-d817-4dde-bb41-b50a8fd3d544-kube-api-access-4j2t7\") pod \"insights-operator-76bcd6ccf5-dwmlf\" (UID: \"1ba9dc85-d817-4dde-bb41-b50a8fd3d544\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.436890 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.436867    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfzfn\" (UniqueName: \"kubernetes.io/projected/bd18742f-fe5e-4fcb-b416-b566be8344e4-kube-api-access-zfzfn\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:01:54.482874 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.482825    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-s5rlp"
Sep 08 10:01:54.516367 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.516343    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5h7jm"
Sep 08 10:01:54.527685 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.527666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf4lj\" (UniqueName: \"kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj\") pod \"network-check-target-pgp7f\" (UID: \"d8cad0d3-0f69-491d-a066-2718f066d04d\") " pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:54.529896 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.529877    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf4lj\" (UniqueName: \"kubernetes.io/projected/d8cad0d3-0f69-491d-a066-2718f066d04d-kube-api-access-hf4lj\") pod \"network-check-target-pgp7f\" (UID: \"d8cad0d3-0f69-491d-a066-2718f066d04d\") " pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:54.549624 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.549604    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf"
Sep 08 10:01:54.636993 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.636277    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:54.641787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.640877    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v9j5d\""
Sep 08 10:01:54.651193 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.651000    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:01:54.730161 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.730130    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:54.730633 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.730608    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 10:01:54.730715 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.730636    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d946c986b-wvnk5: secret "image-registry-tls" not found
Sep 08 10:01:54.730715 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.730703    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls podName:7149cdff-fd81-49d1-9df7-9757dfd544d3 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:55.730680673 +0000 UTC m=+35.571905038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls") pod "image-registry-d946c986b-wvnk5" (UID: "7149cdff-fd81-49d1-9df7-9757dfd544d3") : secret "image-registry-tls" not found
Sep 08 10:01:54.830835 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.830804    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7cwbx\" (UID: \"13d91b33-2bf0-4bf4-b866-8b6bc251c280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx"
Sep 08 10:01:54.830973 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.830874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.830973 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.830918    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-5p6mp\" (UID: \"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"
Sep 08 10:01:54.830973 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.830950    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 10:01:54.830973 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.830963    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:54.831148 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.831009    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert podName:13d91b33-2bf0-4bf4-b866-8b6bc251c280 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:55.830993285 +0000 UTC m=+35.672217646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-7cwbx" (UID: "13d91b33-2bf0-4bf4-b866-8b6bc251c280") : secret "networking-console-plugin-cert" not found
Sep 08 10:01:54.831148 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.831013    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 10:01:54.831148 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.831045    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 10:01:54.831148 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.831066    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs podName:132efab2-1cc6-4fac-a9ef-61d9d8ab4c48 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:55.831052816 +0000 UTC m=+35.672277201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs") pod "router-default-5fbcc7c578-fzqlh" (UID: "132efab2-1cc6-4fac-a9ef-61d9d8ab4c48") : secret "router-metrics-certs-default" not found
Sep 08 10:01:54.831148 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.831081    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle podName:132efab2-1cc6-4fac-a9ef-61d9d8ab4c48 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:55.831074802 +0000 UTC m=+35.672299162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle") pod "router-default-5fbcc7c578-fzqlh" (UID: "132efab2-1cc6-4fac-a9ef-61d9d8ab4c48") : configmap references non-existent config key: service-ca.crt
Sep 08 10:01:54.831148 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.831091    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls podName:e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:55.831085996 +0000 UTC m=+35.672310357 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-5p6mp" (UID: "e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464") : secret "samples-operator-tls" not found
Sep 08 10:01:54.932229 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.932207    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:54.932395 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.932280    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:01:54.932395 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.932330    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 10:01:54.932395 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.932380    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls podName:cacaccdd-dd0d-4bcd-a974-a065758240f7 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:55.932367098 +0000 UTC m=+35.773591473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls") pod "dns-default-fnmlf" (UID: "cacaccdd-dd0d-4bcd-a974-a065758240f7") : secret "dns-default-metrics-tls" not found
Sep 08 10:01:54.932625 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.932403    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 10:01:54.932625 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:54.932451    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls podName:bd18742f-fe5e-4fcb-b416-b566be8344e4 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:55.932435085 +0000 UTC m=+35.773659460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wgqz8" (UID: "bd18742f-fe5e-4fcb-b416-b566be8344e4") : secret "cluster-monitoring-operator-tls" not found
Sep 08 10:01:54.938551 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.936844    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c"]
Sep 08 10:01:54.939392 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.939273    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-s5rlp"]
Sep 08 10:01:54.942395 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.942144    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5h7jm"]
Sep 08 10:01:54.945587 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.945530    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-pm7xz"]
Sep 08 10:01:54.946034 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:54.945995    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2461e11a_d91a_4e42_bb9e_8646646fdf40.slice/crio-e12d4c50da0d86e6ffea2f80608078f553d3f7b20c7262b4abb2363819319333 WatchSource:0}: Error finding container e12d4c50da0d86e6ffea2f80608078f553d3f7b20c7262b4abb2363819319333: Status 404 returned error can't find the container with id e12d4c50da0d86e6ffea2f80608078f553d3f7b20c7262b4abb2363819319333
Sep 08 10:01:54.947354 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.946746    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q"]
Sep 08 10:01:54.947354 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:54.946784    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c33bab_504d_48c2_96f0_5ba84ff76f8f.slice/crio-06c948751aa6cdc94eb6f1f05d5b2ceb9469e1bb1c6f1f50c637bcde6948c50a WatchSource:0}: Error finding container 06c948751aa6cdc94eb6f1f05d5b2ceb9469e1bb1c6f1f50c637bcde6948c50a: Status 404 returned error can't find the container with id 06c948751aa6cdc94eb6f1f05d5b2ceb9469e1bb1c6f1f50c637bcde6948c50a
Sep 08 10:01:54.948105 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:54.947980    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e917bce_6da7_4d7a_bbb2_bd09d8086599.slice/crio-ac0b7f8e11f90f82fe00eace59d52b8fe19700c4b771d0cc3212dae7837eaaba WatchSource:0}: Error finding container ac0b7f8e11f90f82fe00eace59d52b8fe19700c4b771d0cc3212dae7837eaaba: Status 404 returned error can't find the container with id ac0b7f8e11f90f82fe00eace59d52b8fe19700c4b771d0cc3212dae7837eaaba
Sep 08 10:01:54.948105 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.948040    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-dwmlf"]
Sep 08 10:01:54.949669 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:54.949413    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eba85f8_a8d6_4c6b_ad15_bbf9e20e11f9.slice/crio-894278bbf20db31f62a6f69edd53fd5c7959ee0c94b6c2a77dd12d9ccb7a645a WatchSource:0}: Error finding container 894278bbf20db31f62a6f69edd53fd5c7959ee0c94b6c2a77dd12d9ccb7a645a: Status 404 returned error can't find the container with id 894278bbf20db31f62a6f69edd53fd5c7959ee0c94b6c2a77dd12d9ccb7a645a
Sep 08 10:01:54.950350 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:54.950263    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba9dc85_d817_4dde_bb41_b50a8fd3d544.slice/crio-7344035a37570aa2a5aa8ac4ade93f57284ad3aed4a0e6d27a85241868ea412c WatchSource:0}: Error finding container 7344035a37570aa2a5aa8ac4ade93f57284ad3aed4a0e6d27a85241868ea412c: Status 404 returned error can't find the container with id 7344035a37570aa2a5aa8ac4ade93f57284ad3aed4a0e6d27a85241868ea412c
Sep 08 10:01:54.972073 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.972050    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-hpbwn"]
Sep 08 10:01:54.975346 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:54.975220    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pgp7f"]
Sep 08 10:01:54.976045 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:54.976017    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e7a59c_e003_45d4_8878_63728f83b9dc.slice/crio-e5c7f4f5ede4d47b66d4c96570f06f45ac1cba3f882a89644f404615aa2ebbd0 WatchSource:0}: Error finding container e5c7f4f5ede4d47b66d4c96570f06f45ac1cba3f882a89644f404615aa2ebbd0: Status 404 returned error can't find the container with id e5c7f4f5ede4d47b66d4c96570f06f45ac1cba3f882a89644f404615aa2ebbd0
Sep 08 10:01:54.978406 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:01:54.978376    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8cad0d3_0f69_491d_a066_2718f066d04d.slice/crio-8c90fa160abf3c980667fae1796ac855aa8e21af38284891afecc8c8c0e095d7 WatchSource:0}: Error finding container 8c90fa160abf3c980667fae1796ac855aa8e21af38284891afecc8c8c0e095d7: Status 404 returned error can't find the container with id 8c90fa160abf3c980667fae1796ac855aa8e21af38284891afecc8c8c0e095d7
Sep 08 10:01:55.630680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.630354    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:01:55.630680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.630393    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:01:55.634498 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.634295    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 10:01:55.634498 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.634301    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 10:01:55.634498 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.634375    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8nq8l\""
Sep 08 10:01:55.741064 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.739990    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:55.741064 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:55.740184    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 10:01:55.741064 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:55.740200    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d946c986b-wvnk5: secret "image-registry-tls" not found
Sep 08 10:01:55.741064 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:55.740280    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls podName:7149cdff-fd81-49d1-9df7-9757dfd544d3 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:57.740259257 +0000 UTC m=+37.581483633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls") pod "image-registry-d946c986b-wvnk5" (UID: "7149cdff-fd81-49d1-9df7-9757dfd544d3") : secret "image-registry-tls" not found
Sep 08 10:01:55.754952 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.754864    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-s5rlp" event={"ID":"65c33bab-504d-48c2-96f0-5ba84ff76f8f","Type":"ContainerStarted","Data":"06c948751aa6cdc94eb6f1f05d5b2ceb9469e1bb1c6f1f50c637bcde6948c50a"}
Sep 08 10:01:55.757045 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.756985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q" event={"ID":"5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9","Type":"ContainerStarted","Data":"894278bbf20db31f62a6f69edd53fd5c7959ee0c94b6c2a77dd12d9ccb7a645a"}
Sep 08 10:01:55.764718 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.764690    2559 generic.go:356] "Generic (PLEG): container finished" podID="59f5177b-f1ce-4ef1-b7c2-752e58d5e27b" containerID="2a79f3bade352d8b6e6cf4aba94a57e0cc3f83a7c856c5fb2b7d5e140674eddb" exitCode=0
Sep 08 10:01:55.764835 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.764780    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jxq2" event={"ID":"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b","Type":"ContainerDied","Data":"2a79f3bade352d8b6e6cf4aba94a57e0cc3f83a7c856c5fb2b7d5e140674eddb"}
Sep 08 10:01:55.767861 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.767820    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5h7jm" event={"ID":"2461e11a-d91a-4e42-bb9e-8646646fdf40","Type":"ContainerStarted","Data":"e12d4c50da0d86e6ffea2f80608078f553d3f7b20c7262b4abb2363819319333"}
Sep 08 10:01:55.772998 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.772901    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf" event={"ID":"1ba9dc85-d817-4dde-bb41-b50a8fd3d544","Type":"ContainerStarted","Data":"7344035a37570aa2a5aa8ac4ade93f57284ad3aed4a0e6d27a85241868ea412c"}
Sep 08 10:01:55.774956 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.774927    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-hpbwn" event={"ID":"15e7a59c-e003-45d4-8878-63728f83b9dc","Type":"ContainerStarted","Data":"e5c7f4f5ede4d47b66d4c96570f06f45ac1cba3f882a89644f404615aa2ebbd0"}
Sep 08 10:01:55.777190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.777147    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-pm7xz" event={"ID":"4e917bce-6da7-4d7a-bbb2-bd09d8086599","Type":"ContainerStarted","Data":"ac0b7f8e11f90f82fe00eace59d52b8fe19700c4b771d0cc3212dae7837eaaba"}
Sep 08 10:01:55.779906 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.779883    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c" event={"ID":"633deb4b-3d4a-47c5-b6ef-a8d8a803c521","Type":"ContainerStarted","Data":"c1e7a7d94b90cc5261ed16f8ba788e94ba43aec8f902424aeaa436003fee094e"}
Sep 08 10:01:55.782746 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.782724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pgp7f" event={"ID":"d8cad0d3-0f69-491d-a066-2718f066d04d","Type":"ContainerStarted","Data":"8c90fa160abf3c980667fae1796ac855aa8e21af38284891afecc8c8c0e095d7"}
Sep 08 10:01:55.840890 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.840862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7cwbx\" (UID: \"13d91b33-2bf0-4bf4-b866-8b6bc251c280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx"
Sep 08 10:01:55.841012 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.840943    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:55.841012 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.840992    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-5p6mp\" (UID: \"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"
Sep 08 10:01:55.841160 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.841029    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:55.842640 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:55.841217    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle podName:132efab2-1cc6-4fac-a9ef-61d9d8ab4c48 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:57.84119842 +0000 UTC m=+37.682422801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle") pod "router-default-5fbcc7c578-fzqlh" (UID: "132efab2-1cc6-4fac-a9ef-61d9d8ab4c48") : configmap references non-existent config key: service-ca.crt
Sep 08 10:01:55.842640 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:55.841655    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 10:01:55.842640 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:55.841720    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert podName:13d91b33-2bf0-4bf4-b866-8b6bc251c280 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:57.841702361 +0000 UTC m=+37.682926736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-7cwbx" (UID: "13d91b33-2bf0-4bf4-b866-8b6bc251c280") : secret "networking-console-plugin-cert" not found
Sep 08 10:01:55.842640 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:55.842362    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 10:01:55.842640 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:55.842458    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs podName:132efab2-1cc6-4fac-a9ef-61d9d8ab4c48 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:57.842442161 +0000 UTC m=+37.683666525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs") pod "router-default-5fbcc7c578-fzqlh" (UID: "132efab2-1cc6-4fac-a9ef-61d9d8ab4c48") : secret "router-metrics-certs-default" not found
Sep 08 10:01:55.842640 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:55.842520    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 10:01:55.842640 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:55.842555    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls podName:e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:57.842543925 +0000 UTC m=+37.683768299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-5p6mp" (UID: "e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464") : secret "samples-operator-tls" not found
Sep 08 10:01:55.941840 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.941806    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:55.941968 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:55.941876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:01:55.943862 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:55.942842    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 10:01:55.943862 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:55.942899    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls podName:cacaccdd-dd0d-4bcd-a974-a065758240f7 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:57.94288207 +0000 UTC m=+37.784106432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls") pod "dns-default-fnmlf" (UID: "cacaccdd-dd0d-4bcd-a974-a065758240f7") : secret "dns-default-metrics-tls" not found
Sep 08 10:01:55.943862 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:55.943282    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 10:01:55.943862 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:55.943343    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls podName:bd18742f-fe5e-4fcb-b416-b566be8344e4 nodeName:}" failed. No retries permitted until 2026-09-08 10:01:57.943326564 +0000 UTC m=+37.784550927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wgqz8" (UID: "bd18742f-fe5e-4fcb-b416-b566be8344e4") : secret "cluster-monitoring-operator-tls" not found
Sep 08 10:01:56.790210 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:56.790113    2559 generic.go:356] "Generic (PLEG): container finished" podID="59f5177b-f1ce-4ef1-b7c2-752e58d5e27b" containerID="a9e04fc679567c52f9c138273ea43c77b7458bf54ff108b30f8e351e3ec6f57e" exitCode=0
Sep 08 10:01:56.790210 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:56.790195    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jxq2" event={"ID":"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b","Type":"ContainerDied","Data":"a9e04fc679567c52f9c138273ea43c77b7458bf54ff108b30f8e351e3ec6f57e"}
Sep 08 10:01:57.762950 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:57.762906    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:01:57.763179 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:57.763155    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 10:01:57.763179 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:57.763177    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d946c986b-wvnk5: secret "image-registry-tls" not found
Sep 08 10:01:57.763409 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:57.763274    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls podName:7149cdff-fd81-49d1-9df7-9757dfd544d3 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:01.763223281 +0000 UTC m=+41.604447662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls") pod "image-registry-d946c986b-wvnk5" (UID: "7149cdff-fd81-49d1-9df7-9757dfd544d3") : secret "image-registry-tls" not found
Sep 08 10:01:57.864183 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:57.864136    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7cwbx\" (UID: \"13d91b33-2bf0-4bf4-b866-8b6bc251c280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx"
Sep 08 10:01:57.864974 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:57.864223    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:57.864974 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:57.864320    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 10:01:57.864974 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:57.864357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-5p6mp\" (UID: \"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"
Sep 08 10:01:57.864974 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:57.864369    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 10:01:57.864974 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:57.864400    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert podName:13d91b33-2bf0-4bf4-b866-8b6bc251c280 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:01.864376615 +0000 UTC m=+41.705601007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-7cwbx" (UID: "13d91b33-2bf0-4bf4-b866-8b6bc251c280") : secret "networking-console-plugin-cert" not found
Sep 08 10:01:57.864974 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:57.864432    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs podName:132efab2-1cc6-4fac-a9ef-61d9d8ab4c48 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:01.864413247 +0000 UTC m=+41.705637632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs") pod "router-default-5fbcc7c578-fzqlh" (UID: "132efab2-1cc6-4fac-a9ef-61d9d8ab4c48") : secret "router-metrics-certs-default" not found
Sep 08 10:01:57.864974 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:57.864444    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 10:01:57.864974 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:57.864478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:01:57.864974 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:57.864488    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls podName:e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:01.864473334 +0000 UTC m=+41.705697708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-5p6mp" (UID: "e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464") : secret "samples-operator-tls" not found
Sep 08 10:01:57.864974 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:57.864589    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle podName:132efab2-1cc6-4fac-a9ef-61d9d8ab4c48 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:01.864576477 +0000 UTC m=+41.705800856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle") pod "router-default-5fbcc7c578-fzqlh" (UID: "132efab2-1cc6-4fac-a9ef-61d9d8ab4c48") : configmap references non-existent config key: service-ca.crt
Sep 08 10:01:57.966592 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:57.965609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:01:57.966592 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:01:57.965671    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:01:57.966592 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:57.965895    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 10:01:57.966592 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:57.965943    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls podName:bd18742f-fe5e-4fcb-b416-b566be8344e4 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:01.965930169 +0000 UTC m=+41.807154530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wgqz8" (UID: "bd18742f-fe5e-4fcb-b416-b566be8344e4") : secret "cluster-monitoring-operator-tls" not found
Sep 08 10:01:57.966592 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:57.966396    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 10:01:57.966592 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:01:57.966457    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls podName:cacaccdd-dd0d-4bcd-a974-a065758240f7 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:01.966438117 +0000 UTC m=+41.807662501 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls") pod "dns-default-fnmlf" (UID: "cacaccdd-dd0d-4bcd-a974-a065758240f7") : secret "dns-default-metrics-tls" not found
Sep 08 10:02:01.803513 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.803400    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:02:01.804055 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:01.803710    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 10:02:01.804055 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:01.803727    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d946c986b-wvnk5: secret "image-registry-tls" not found
Sep 08 10:02:01.804055 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:01.803785    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls podName:7149cdff-fd81-49d1-9df7-9757dfd544d3 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:09.803763938 +0000 UTC m=+49.644988299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls") pod "image-registry-d946c986b-wvnk5" (UID: "7149cdff-fd81-49d1-9df7-9757dfd544d3") : secret "image-registry-tls" not found
Sep 08 10:02:01.811103 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.810557    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-s5rlp" event={"ID":"65c33bab-504d-48c2-96f0-5ba84ff76f8f","Type":"ContainerStarted","Data":"629491846c8979df73dbef818e246aed7313a0c87a5e3cd31b733501c644caa5"}
Sep 08 10:02:01.813506 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.812952    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q" event={"ID":"5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9","Type":"ContainerStarted","Data":"5f5833125410b72aa65f21863df6b408d943b8d2c2a5458827c9d87a19252365"}
Sep 08 10:02:01.819731 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.819707    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jxq2" event={"ID":"59f5177b-f1ce-4ef1-b7c2-752e58d5e27b","Type":"ContainerStarted","Data":"2c74ee3fefc16f50b6463b1ba54df75af067e0ab064fa5d5e0dcc9a4ee6ee60b"}
Sep 08 10:02:01.821834 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.821809    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5h7jm" event={"ID":"2461e11a-d91a-4e42-bb9e-8646646fdf40","Type":"ContainerStarted","Data":"5b569794dc28fbdb6102292b01069fd17ccc230f23bb5776bbc952872f8df97f"}
Sep 08 10:02:01.825356 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.825327    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-hpbwn" event={"ID":"15e7a59c-e003-45d4-8878-63728f83b9dc","Type":"ContainerStarted","Data":"1b7e23fd2c781a725bb473283c6c70294d6bd25810091190005eac017788fec0"}
Sep 08 10:02:01.826322 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.826276    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-s5rlp" podStartSLOduration=22.55616795 podStartE2EDuration="28.826234909s" totalImagesPullingTime="6.270066959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:01.825850137 +0000 UTC m=+41.667074522" watchObservedRunningTime="2026-09-08 10:02:01.826234909 +0000 UTC m=+41.667459286"
Sep 08 10:02:01.829883 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.829844    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c" event={"ID":"633deb4b-3d4a-47c5-b6ef-a8d8a803c521","Type":"ContainerStarted","Data":"c3a604a358912c34c4bef492bc3accab9a48ab7a8cebf9a3ffa5bc555303181b"}
Sep 08 10:02:01.833292 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.833268    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pgp7f" event={"ID":"d8cad0d3-0f69-491d-a066-2718f066d04d","Type":"ContainerStarted","Data":"8aab9cecf77097b476d0c663bd53f7d26d55a35f0aada294b11aa289ed999de5"}
Sep 08 10:02:01.833703 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.833686    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:02:01.845371 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.845327    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5h7jm" podStartSLOduration=2.793308394 podStartE2EDuration="8.845311658s" totalImagesPullingTime="6.052003264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:01.843314253 +0000 UTC m=+41.684538663" watchObservedRunningTime="2026-09-08 10:02:01.845311658 +0000 UTC m=+41.686536047"
Sep 08 10:02:01.886169 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.885605    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4jxq2" podStartSLOduration=15.548578071 podStartE2EDuration="40.885587074s" totalImagesPullingTime="25.337009003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:01.884445348 +0000 UTC m=+41.725669752" watchObservedRunningTime="2026-09-08 10:02:01.885587074 +0000 UTC m=+41.726811458"
Sep 08 10:02:01.886169 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.885993    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-hpbwn" podStartSLOduration=22.925438203 podStartE2EDuration="28.885985448s" totalImagesPullingTime="5.960547245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:01.86066971 +0000 UTC m=+41.701894116" watchObservedRunningTime="2026-09-08 10:02:01.885985448 +0000 UTC m=+41.727209831"
Sep 08 10:02:01.904016 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.903944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-5p6mp\" (UID: \"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"
Sep 08 10:02:01.904016 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.903993    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:02:01.904172 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.904102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7cwbx\" (UID: \"13d91b33-2bf0-4bf4-b866-8b6bc251c280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx"
Sep 08 10:02:01.904172 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.904163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:02:01.904387 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:01.904306    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 10:02:01.904447 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:01.904429    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 10:02:01.904498 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:01.904454    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs podName:132efab2-1cc6-4fac-a9ef-61d9d8ab4c48 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:09.904432451 +0000 UTC m=+49.745656835 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs") pod "router-default-5fbcc7c578-fzqlh" (UID: "132efab2-1cc6-4fac-a9ef-61d9d8ab4c48") : secret "router-metrics-certs-default" not found
Sep 08 10:02:01.904750 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.904657    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q" podStartSLOduration=22.855845575 podStartE2EDuration="28.904641725s" totalImagesPullingTime="6.04879615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:01.9035614 +0000 UTC m=+41.744785804" watchObservedRunningTime="2026-09-08 10:02:01.904641725 +0000 UTC m=+41.745866109"
Sep 08 10:02:01.904750 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:01.904715    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls podName:e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:09.90468983 +0000 UTC m=+49.745914194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-5p6mp" (UID: "e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464") : secret "samples-operator-tls" not found
Sep 08 10:02:01.904906 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:01.904759    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 10:02:01.904906 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:01.904808    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle podName:132efab2-1cc6-4fac-a9ef-61d9d8ab4c48 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:09.904795138 +0000 UTC m=+49.746019520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle") pod "router-default-5fbcc7c578-fzqlh" (UID: "132efab2-1cc6-4fac-a9ef-61d9d8ab4c48") : configmap references non-existent config key: service-ca.crt
Sep 08 10:02:01.904906 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:01.904864    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert podName:13d91b33-2bf0-4bf4-b866-8b6bc251c280 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:09.904848974 +0000 UTC m=+49.746073358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-7cwbx" (UID: "13d91b33-2bf0-4bf4-b866-8b6bc251c280") : secret "networking-console-plugin-cert" not found
Sep 08 10:02:01.926458 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:01.925605    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c" podStartSLOduration=22.871358364 podStartE2EDuration="28.92558697s" totalImagesPullingTime="6.054228606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:01.9245212 +0000 UTC m=+41.765745610" watchObservedRunningTime="2026-09-08 10:02:01.92558697 +0000 UTC m=+41.766811355"
Sep 08 10:02:02.015481 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.014098    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:02:02.015481 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.014548    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:02:02.015481 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:02.014833    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 10:02:02.015481 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:02.014888    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls podName:bd18742f-fe5e-4fcb-b416-b566be8344e4 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:10.014871129 +0000 UTC m=+49.856095498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wgqz8" (UID: "bd18742f-fe5e-4fcb-b416-b566be8344e4") : secret "cluster-monitoring-operator-tls" not found
Sep 08 10:02:02.015481 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:02.014927    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 10:02:02.015481 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:02.014975    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls podName:cacaccdd-dd0d-4bcd-a974-a065758240f7 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:10.014959986 +0000 UTC m=+49.856184376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls") pod "dns-default-fnmlf" (UID: "cacaccdd-dd0d-4bcd-a974-a065758240f7") : secret "dns-default-metrics-tls" not found
Sep 08 10:02:02.402339 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.402270    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pgp7f" podStartSLOduration=35.371417965 podStartE2EDuration="41.402217391s" totalImagesPullingTime="6.030799426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:01.94679217 +0000 UTC m=+41.788016554" watchObservedRunningTime="2026-09-08 10:02:02.402217391 +0000 UTC m=+42.243441769"
Sep 08 10:02:02.403901 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.403517    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9r4k7"]
Sep 08 10:02:02.420475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.420449    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9r4k7"
Sep 08 10:02:02.421598 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.421574    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9r4k7"]
Sep 08 10:02:02.423723 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.423701    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 10:02:02.424114 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.424094    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nqchb\""
Sep 08 10:02:02.424410 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.424338    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 10:02:02.521045 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.521018    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsmw2\" (UniqueName: \"kubernetes.io/projected/921284cb-1a60-4623-80bf-e2c32870de7d-kube-api-access-lsmw2\") pod \"migrator-fdbb8fdff-9r4k7\" (UID: \"921284cb-1a60-4623-80bf-e2c32870de7d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9r4k7"
Sep 08 10:02:02.621877 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.621835    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsmw2\" (UniqueName: \"kubernetes.io/projected/921284cb-1a60-4623-80bf-e2c32870de7d-kube-api-access-lsmw2\") pod \"migrator-fdbb8fdff-9r4k7\" (UID: \"921284cb-1a60-4623-80bf-e2c32870de7d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9r4k7"
Sep 08 10:02:02.631809 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.631780    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsmw2\" (UniqueName: \"kubernetes.io/projected/921284cb-1a60-4623-80bf-e2c32870de7d-kube-api-access-lsmw2\") pod \"migrator-fdbb8fdff-9r4k7\" (UID: \"921284cb-1a60-4623-80bf-e2c32870de7d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9r4k7"
Sep 08 10:02:02.733257 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.733186    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9r4k7"
Sep 08 10:02:02.845088 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.845045    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf" event={"ID":"1ba9dc85-d817-4dde-bb41-b50a8fd3d544","Type":"ContainerStarted","Data":"96f07faf674b2c1389ed0919079aeb0e8a9c206c54348fe6921675fc3cc5f37f"}
Sep 08 10:02:02.866430 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:02.866381    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf" podStartSLOduration=22.336253988 podStartE2EDuration="29.866361443s" totalImagesPullingTime="7.530107455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:02.864838929 +0000 UTC m=+42.706063325" watchObservedRunningTime="2026-09-08 10:02:02.866361443 +0000 UTC m=+42.707585830"
Sep 08 10:02:03.606379 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:03.606353    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9r4k7"]
Sep 08 10:02:03.667781 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:02:03.667754    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921284cb_1a60_4623_80bf_e2c32870de7d.slice/crio-0e49291acadcac5cf4e37ed9a2e52e353ac17726d7e621a70873a3bfbcca2ca8 WatchSource:0}: Error finding container 0e49291acadcac5cf4e37ed9a2e52e353ac17726d7e621a70873a3bfbcca2ca8: Status 404 returned error can't find the container with id 0e49291acadcac5cf4e37ed9a2e52e353ac17726d7e621a70873a3bfbcca2ca8
Sep 08 10:02:03.849538 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:03.849507    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9r4k7" event={"ID":"921284cb-1a60-4623-80bf-e2c32870de7d","Type":"ContainerStarted","Data":"0e49291acadcac5cf4e37ed9a2e52e353ac17726d7e621a70873a3bfbcca2ca8"}
Sep 08 10:02:03.850967 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:03.850935    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-pm7xz" event={"ID":"4e917bce-6da7-4d7a-bbb2-bd09d8086599","Type":"ContainerStarted","Data":"a48a7c1ba28bd6bca5f5a877774176f4d4a880583a9dd8d78af5b693d95c44fb"}
Sep 08 10:02:04.247956 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.247906    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-pm7xz" podStartSLOduration=22.5021748 podStartE2EDuration="31.24789077s" totalImagesPullingTime="8.74571597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:03.867948143 +0000 UTC m=+43.709172554" watchObservedRunningTime="2026-09-08 10:02:04.24789077 +0000 UTC m=+44.089115170"
Sep 08 10:02:04.248797 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.248778    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wvgzg"]
Sep 08 10:02:04.270875 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.270850    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wvgzg"]
Sep 08 10:02:04.270988 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.270948    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.273929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.273903    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 10:02:04.274049 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.273906    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 10:02:04.274049 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.273931    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hrpwv\""
Sep 08 10:02:04.337935 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.337907    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-crio-socket\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.338043 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.337952    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-data-volume\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.338043 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.337982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.338135 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.338102    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.338173 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.338141    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vclwm\" (UniqueName: \"kubernetes.io/projected/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-api-access-vclwm\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.404717 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.404685    2559 patch_prober.go:28] interesting pod/console-operator-85b454874b-pm7xz container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.13:8443/healthz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body=
Sep 08 10:02:04.404829 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.404733    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-85b454874b-pm7xz" podUID="4e917bce-6da7-4d7a-bbb2-bd09d8086599" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/healthz\": dial tcp 10.132.0.13:8443: connect: connection refused"
Sep 08 10:02:04.438563 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.438540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-crio-socket\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.438674 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.438576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-data-volume\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.438674 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.438609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.438674 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.438661    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.438818 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.438696    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vclwm\" (UniqueName: \"kubernetes.io/projected/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-api-access-vclwm\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.438818 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:04.438750    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 10:02:04.438818 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.438789    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-crio-socket\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.438818 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:04.438802    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm podName:d251d8ad-8bef-4a17-8cd3-7da32a35deb6 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:04.938787985 +0000 UTC m=+44.780012346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wvgzg" (UID: "d251d8ad-8bef-4a17-8cd3-7da32a35deb6") : configmap "kube-rbac-proxy" not found
Sep 08 10:02:04.438818 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:04.438803    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 10:02:04.439015 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:04.438866    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls podName:d251d8ad-8bef-4a17-8cd3-7da32a35deb6 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:04.938847777 +0000 UTC m=+44.780072152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvgzg" (UID: "d251d8ad-8bef-4a17-8cd3-7da32a35deb6") : secret "insights-runtime-extractor-tls" not found
Sep 08 10:02:04.439015 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.438879    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-data-volume\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.446929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.446914    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vclwm\" (UniqueName: \"kubernetes.io/projected/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-api-access-vclwm\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.855030 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.855004    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/0.log"
Sep 08 10:02:04.855428 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.855042    2559 generic.go:356] "Generic (PLEG): container finished" podID="4e917bce-6da7-4d7a-bbb2-bd09d8086599" containerID="a48a7c1ba28bd6bca5f5a877774176f4d4a880583a9dd8d78af5b693d95c44fb" exitCode=255
Sep 08 10:02:04.855428 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.855072    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-pm7xz" event={"ID":"4e917bce-6da7-4d7a-bbb2-bd09d8086599","Type":"ContainerDied","Data":"a48a7c1ba28bd6bca5f5a877774176f4d4a880583a9dd8d78af5b693d95c44fb"}
Sep 08 10:02:04.855428 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.855339    2559 scope.go:117] "RemoveContainer" containerID="a48a7c1ba28bd6bca5f5a877774176f4d4a880583a9dd8d78af5b693d95c44fb"
Sep 08 10:02:04.943487 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.943465    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.943597 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:04.943574    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:04.943654 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:04.943602    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 10:02:04.943749 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:04.943731    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 10:02:04.943822 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:04.943809    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm podName:d251d8ad-8bef-4a17-8cd3-7da32a35deb6 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:05.943788349 +0000 UTC m=+45.785012726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wvgzg" (UID: "d251d8ad-8bef-4a17-8cd3-7da32a35deb6") : configmap "kube-rbac-proxy" not found
Sep 08 10:02:04.943871 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:04.943836    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls podName:d251d8ad-8bef-4a17-8cd3-7da32a35deb6 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:05.943829771 +0000 UTC m=+45.785054132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvgzg" (UID: "d251d8ad-8bef-4a17-8cd3-7da32a35deb6") : secret "insights-runtime-extractor-tls" not found
Sep 08 10:02:05.859566 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:05.859543    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:02:05.860002 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:05.859985    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/0.log"
Sep 08 10:02:05.860064 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:05.860019    2559 generic.go:356] "Generic (PLEG): container finished" podID="4e917bce-6da7-4d7a-bbb2-bd09d8086599" containerID="65a0fca29ce48245c4c5a8014910001068d9a2c7abb64afce5e7c60f1f5d94cb" exitCode=255
Sep 08 10:02:05.860064 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:05.860052    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-pm7xz" event={"ID":"4e917bce-6da7-4d7a-bbb2-bd09d8086599","Type":"ContainerDied","Data":"65a0fca29ce48245c4c5a8014910001068d9a2c7abb64afce5e7c60f1f5d94cb"}
Sep 08 10:02:05.860155 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:05.860118    2559 scope.go:117] "RemoveContainer" containerID="a48a7c1ba28bd6bca5f5a877774176f4d4a880583a9dd8d78af5b693d95c44fb"
Sep 08 10:02:05.860369 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:05.860346    2559 scope.go:117] "RemoveContainer" containerID="65a0fca29ce48245c4c5a8014910001068d9a2c7abb64afce5e7c60f1f5d94cb"
Sep 08 10:02:05.860546 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:05.860524    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-pm7xz_openshift-console-operator(4e917bce-6da7-4d7a-bbb2-bd09d8086599)\"" pod="openshift-console-operator/console-operator-85b454874b-pm7xz" podUID="4e917bce-6da7-4d7a-bbb2-bd09d8086599"
Sep 08 10:02:05.951306 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:05.951282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:05.951442 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:05.951396    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 10:02:05.951506 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:05.951454    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls podName:d251d8ad-8bef-4a17-8cd3-7da32a35deb6 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:07.951435328 +0000 UTC m=+47.792659691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvgzg" (UID: "d251d8ad-8bef-4a17-8cd3-7da32a35deb6") : secret "insights-runtime-extractor-tls" not found
Sep 08 10:02:05.951581 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:05.951551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:05.951701 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:05.951682    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 10:02:05.951768 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:05.951737    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm podName:d251d8ad-8bef-4a17-8cd3-7da32a35deb6 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:07.951723682 +0000 UTC m=+47.792948048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wvgzg" (UID: "d251d8ad-8bef-4a17-8cd3-7da32a35deb6") : configmap "kube-rbac-proxy" not found
Sep 08 10:02:06.446326 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:06.446303    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5lzjn_ff849db0-791c-405b-bbfc-868f1b7288d2/dns-node-resolver/0.log"
Sep 08 10:02:06.864681 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:06.864615    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:02:06.865015 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:06.864906    2559 scope.go:117] "RemoveContainer" containerID="65a0fca29ce48245c4c5a8014910001068d9a2c7abb64afce5e7c60f1f5d94cb"
Sep 08 10:02:06.865061 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:06.865037    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-pm7xz_openshift-console-operator(4e917bce-6da7-4d7a-bbb2-bd09d8086599)\"" pod="openshift-console-operator/console-operator-85b454874b-pm7xz" podUID="4e917bce-6da7-4d7a-bbb2-bd09d8086599"
Sep 08 10:02:07.430486 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:07.430461    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hp9jt_2719b26a-ad31-4018-a46c-927706ebc26e/node-ca/0.log"
Sep 08 10:02:07.969216 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:07.969191    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:07.969573 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:07.969270    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:07.969573 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:07.969310    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 10:02:07.969573 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:07.969377    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm podName:d251d8ad-8bef-4a17-8cd3-7da32a35deb6 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:11.96936159 +0000 UTC m=+51.810585962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wvgzg" (UID: "d251d8ad-8bef-4a17-8cd3-7da32a35deb6") : configmap "kube-rbac-proxy" not found
Sep 08 10:02:07.969573 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:07.969405    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 10:02:07.969573 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:07.969456    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls podName:d251d8ad-8bef-4a17-8cd3-7da32a35deb6 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:11.969443259 +0000 UTC m=+51.810667639 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvgzg" (UID: "d251d8ad-8bef-4a17-8cd3-7da32a35deb6") : secret "insights-runtime-extractor-tls" not found
Sep 08 10:02:08.230908 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:08.230887    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5h7jm_2461e11a-d91a-4e42-bb9e-8646646fdf40/serve-healthcheck-canary/0.log"
Sep 08 10:02:08.876316 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:08.876285    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9r4k7" event={"ID":"921284cb-1a60-4623-80bf-e2c32870de7d","Type":"ContainerStarted","Data":"7a54b74aab29544ff0b668e88b5b24fe87c28eb60cff1912b78445e54283bbd5"}
Sep 08 10:02:08.876489 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:08.876334    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9r4k7" event={"ID":"921284cb-1a60-4623-80bf-e2c32870de7d","Type":"ContainerStarted","Data":"8e32b824dbdc92be2cd93c1e4dec48f70b4552621d2f592790c579b43eb70dde"}
Sep 08 10:02:08.895409 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:08.895363    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9r4k7" podStartSLOduration=2.539667811 podStartE2EDuration="6.89534814s" totalImagesPullingTime="4.355680329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:08.894986619 +0000 UTC m=+48.736211022" watchObservedRunningTime="2026-09-08 10:02:08.89534814 +0000 UTC m=+48.736572523"
Sep 08 10:02:09.033051 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:09.033022    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-ls69q_5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9/kube-storage-version-migrator-operator/0.log"
Sep 08 10:02:09.884814 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:09.884786    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:02:09.884986 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:09.884939    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 10:02:09.884986 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:09.884957    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d946c986b-wvnk5: secret "image-registry-tls" not found
Sep 08 10:02:09.885104 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:09.885008    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls podName:7149cdff-fd81-49d1-9df7-9757dfd544d3 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:25.884994128 +0000 UTC m=+65.726218489 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls") pod "image-registry-d946c986b-wvnk5" (UID: "7149cdff-fd81-49d1-9df7-9757dfd544d3") : secret "image-registry-tls" not found
Sep 08 10:02:09.985290 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:09.985263    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7cwbx\" (UID: \"13d91b33-2bf0-4bf4-b866-8b6bc251c280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx"
Sep 08 10:02:09.985395 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:09.985313    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:02:09.985395 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:09.985354    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-5p6mp\" (UID: \"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"
Sep 08 10:02:09.985395 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:09.985372    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 10:02:09.985538 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:09.985375    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:02:09.985538 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:09.985440    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert podName:13d91b33-2bf0-4bf4-b866-8b6bc251c280 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:25.985420329 +0000 UTC m=+65.826644706 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-7cwbx" (UID: "13d91b33-2bf0-4bf4-b866-8b6bc251c280") : secret "networking-console-plugin-cert" not found
Sep 08 10:02:09.985538 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:09.985440    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 10:02:09.985538 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:09.985465    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle podName:132efab2-1cc6-4fac-a9ef-61d9d8ab4c48 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:25.985451947 +0000 UTC m=+65.826676327 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle") pod "router-default-5fbcc7c578-fzqlh" (UID: "132efab2-1cc6-4fac-a9ef-61d9d8ab4c48") : configmap references non-existent config key: service-ca.crt
Sep 08 10:02:09.985538 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:09.985486    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs podName:132efab2-1cc6-4fac-a9ef-61d9d8ab4c48 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:25.985476436 +0000 UTC m=+65.826700797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs") pod "router-default-5fbcc7c578-fzqlh" (UID: "132efab2-1cc6-4fac-a9ef-61d9d8ab4c48") : secret "router-metrics-certs-default" not found
Sep 08 10:02:09.985538 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:09.985496    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 10:02:09.985820 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:09.985553    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls podName:e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:25.985540371 +0000 UTC m=+65.826764746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-5p6mp" (UID: "e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464") : secret "samples-operator-tls" not found
Sep 08 10:02:10.086764 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:10.086736    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:02:10.087100 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:10.086777    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:02:10.087100 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:10.086868    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 10:02:10.087100 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:10.086923    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls podName:cacaccdd-dd0d-4bcd-a974-a065758240f7 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:26.086910662 +0000 UTC m=+65.928135027 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls") pod "dns-default-fnmlf" (UID: "cacaccdd-dd0d-4bcd-a974-a065758240f7") : secret "dns-default-metrics-tls" not found
Sep 08 10:02:10.087100 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:10.086872    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 10:02:10.087100 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:10.087035    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls podName:bd18742f-fe5e-4fcb-b416-b566be8344e4 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:26.087015612 +0000 UTC m=+65.928239974 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wgqz8" (UID: "bd18742f-fe5e-4fcb-b416-b566be8344e4") : secret "cluster-monitoring-operator-tls" not found
Sep 08 10:02:12.000789 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:12.000753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:12.001515 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:12.000899    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 10:02:12.001515 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:12.000992    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:12.001515 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:12.001032    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls podName:d251d8ad-8bef-4a17-8cd3-7da32a35deb6 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:20.001016503 +0000 UTC m=+59.842240876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvgzg" (UID: "d251d8ad-8bef-4a17-8cd3-7da32a35deb6") : secret "insights-runtime-extractor-tls" not found
Sep 08 10:02:12.001707 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:12.001662    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:13.852102 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:13.852073    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:02:13.852562 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:13.852438    2559 scope.go:117] "RemoveContainer" containerID="65a0fca29ce48245c4c5a8014910001068d9a2c7abb64afce5e7c60f1f5d94cb"
Sep 08 10:02:13.852603 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:13.852578    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-pm7xz_openshift-console-operator(4e917bce-6da7-4d7a-bbb2-bd09d8086599)\"" pod="openshift-console-operator/console-operator-85b454874b-pm7xz" podUID="4e917bce-6da7-4d7a-bbb2-bd09d8086599"
Sep 08 10:02:14.403626 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:14.403591    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:02:14.403974 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:14.403961    2559 scope.go:117] "RemoveContainer" containerID="65a0fca29ce48245c4c5a8014910001068d9a2c7abb64afce5e7c60f1f5d94cb"
Sep 08 10:02:14.404121 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:14.404102    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-pm7xz_openshift-console-operator(4e917bce-6da7-4d7a-bbb2-bd09d8086599)\"" pod="openshift-console-operator/console-operator-85b454874b-pm7xz" podUID="4e917bce-6da7-4d7a-bbb2-bd09d8086599"
Sep 08 10:02:19.749377 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:19.749343    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dtdm"
Sep 08 10:02:20.074057 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:20.073971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:20.076513 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:20.076482    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d251d8ad-8bef-4a17-8cd3-7da32a35deb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvgzg\" (UID: \"d251d8ad-8bef-4a17-8cd3-7da32a35deb6\") " pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:20.179414 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:20.179383    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wvgzg"
Sep 08 10:02:20.450369 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:20.450352    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wvgzg"]
Sep 08 10:02:20.452840 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:02:20.452816    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd251d8ad_8bef_4a17_8cd3_7da32a35deb6.slice/crio-14e259ea111d2b3275e10aa5842035baf738a08d1e2c336c8946c79657857b1b WatchSource:0}: Error finding container 14e259ea111d2b3275e10aa5842035baf738a08d1e2c336c8946c79657857b1b: Status 404 returned error can't find the container with id 14e259ea111d2b3275e10aa5842035baf738a08d1e2c336c8946c79657857b1b
Sep 08 10:02:20.911537 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:20.911506    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvgzg" event={"ID":"d251d8ad-8bef-4a17-8cd3-7da32a35deb6","Type":"ContainerStarted","Data":"e557eb9b1b9e6b129ab54242d5e82b13e164c6f6a59b27e4b1d32fb8e6c29413"}
Sep 08 10:02:20.911884 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:20.911569    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvgzg" event={"ID":"d251d8ad-8bef-4a17-8cd3-7da32a35deb6","Type":"ContainerStarted","Data":"14e259ea111d2b3275e10aa5842035baf738a08d1e2c336c8946c79657857b1b"}
Sep 08 10:02:21.916618 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:21.916587    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvgzg" event={"ID":"d251d8ad-8bef-4a17-8cd3-7da32a35deb6","Type":"ContainerStarted","Data":"43b847de0d960417e6925de545edf2e944822359f9636d4192490495278c0876"}
Sep 08 10:02:24.631134 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:24.631104    2559 scope.go:117] "RemoveContainer" containerID="65a0fca29ce48245c4c5a8014910001068d9a2c7abb64afce5e7c60f1f5d94cb"
Sep 08 10:02:24.927407 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:24.927375    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:02:24.927559 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:24.927443    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-pm7xz" event={"ID":"4e917bce-6da7-4d7a-bbb2-bd09d8086599","Type":"ContainerStarted","Data":"211517a4ea09fb09ff736a81379037c415cedcd6521703bef3a2f361b9421f84"}
Sep 08 10:02:24.927744 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:24.927725    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:02:25.125282 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:25.125227    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-pm7xz"
Sep 08 10:02:25.924277 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:25.924220    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:02:25.926644 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:25.926619    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls\") pod \"image-registry-d946c986b-wvnk5\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") " pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:02:26.024846 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.024815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7cwbx\" (UID: \"13d91b33-2bf0-4bf4-b866-8b6bc251c280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx"
Sep 08 10:02:26.024997 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.024860    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:02:26.024997 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.024882    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-5p6mp\" (UID: \"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"
Sep 08 10:02:26.024997 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.024904    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:02:26.025623 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.025591    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-service-ca-bundle\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:02:26.027312 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.027283    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13d91b33-2bf0-4bf4-b866-8b6bc251c280-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-7cwbx\" (UID: \"13d91b33-2bf0-4bf4-b866-8b6bc251c280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx"
Sep 08 10:02:26.027411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.027342    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-5p6mp\" (UID: \"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"
Sep 08 10:02:26.027411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.027363    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132efab2-1cc6-4fac-a9ef-61d9d8ab4c48-metrics-certs\") pod \"router-default-5fbcc7c578-fzqlh\" (UID: \"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48\") " pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:02:26.125646 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.125623    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:02:26.125749 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.125655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:02:26.127722 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.127702    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacaccdd-dd0d-4bcd-a974-a065758240f7-metrics-tls\") pod \"dns-default-fnmlf\" (UID: \"cacaccdd-dd0d-4bcd-a974-a065758240f7\") " pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:02:26.127889 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.127852    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd18742f-fe5e-4fcb-b416-b566be8344e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wgqz8\" (UID: \"bd18742f-fe5e-4fcb-b416-b566be8344e4\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:02:26.144992 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.144968    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9wz2f\""
Sep 08 10:02:26.153634 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.153613    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:02:26.173359 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.173337    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cdwlb\""
Sep 08 10:02:26.181630 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.181581    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:02:26.192651 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.192633    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jjps9\""
Sep 08 10:02:26.200633 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.200616    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx"
Sep 08 10:02:26.254790 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.254306    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c7kg4\""
Sep 08 10:02:26.262738 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.262133    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"
Sep 08 10:02:26.264939 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.264806    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2pfvl\""
Sep 08 10:02:26.273429 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.273405    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"
Sep 08 10:02:26.300928 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.300907    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pd9dz\""
Sep 08 10:02:26.309140 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.309120    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:02:26.330431 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.330405    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:02:26.330864 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.330522    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs\") pod \"network-metrics-daemon-2pgcf\" (UID: \"a8a40235-67e9-4b28-ad0c-97751e2afe10\") " pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:02:26.336718 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.336537    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 10:02:26.337462 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.337430    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 10:02:26.346049 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.345994    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a40235-67e9-4b28-ad0c-97751e2afe10-metrics-certs\") pod \"network-metrics-daemon-2pgcf\" (UID: \"a8a40235-67e9-4b28-ad0c-97751e2afe10\") " pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:02:26.347037 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.347013    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81df3687-05a6-4b77-ac67-2f8c6626e906-original-pull-secret\") pod \"global-pull-secret-syncer-9h7fv\" (UID: \"81df3687-05a6-4b77-ac67-2f8c6626e906\") " pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:02:26.460705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.460620    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d946c986b-wvnk5"]
Sep 08 10:02:26.463299 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:02:26.463275    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7149cdff_fd81_49d1_9df7_9757dfd544d3.slice/crio-d3a99b9f105901f6a450b6e46d8030517b20d3053d1fd798a812dac6071190c8 WatchSource:0}: Error finding container d3a99b9f105901f6a450b6e46d8030517b20d3053d1fd798a812dac6071190c8: Status 404 returned error can't find the container with id d3a99b9f105901f6a450b6e46d8030517b20d3053d1fd798a812dac6071190c8
Sep 08 10:02:26.481572 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.481553    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5fbcc7c578-fzqlh"]
Sep 08 10:02:26.483281 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:02:26.483258    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132efab2_1cc6_4fac_a9ef_61d9d8ab4c48.slice/crio-f6a170988ded7843e117b3a24f740744b085d34916af59203e7e0650a866c20d WatchSource:0}: Error finding container f6a170988ded7843e117b3a24f740744b085d34916af59203e7e0650a866c20d: Status 404 returned error can't find the container with id f6a170988ded7843e117b3a24f740744b085d34916af59203e7e0650a866c20d
Sep 08 10:02:26.501038 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.501009    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-7cwbx"]
Sep 08 10:02:26.503307 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:02:26.503281    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d91b33_2bf0_4bf4_b866_8b6bc251c280.slice/crio-65b412eda8cc94d0c2267ba716ab7dcb09b950d3a493190a3e9e5e0eef15179d WatchSource:0}: Error finding container 65b412eda8cc94d0c2267ba716ab7dcb09b950d3a493190a3e9e5e0eef15179d: Status 404 returned error can't find the container with id 65b412eda8cc94d0c2267ba716ab7dcb09b950d3a493190a3e9e5e0eef15179d
Sep 08 10:02:26.544849 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.544829    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9h7fv"
Sep 08 10:02:26.554065 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.554047    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8nq8l\""
Sep 08 10:02:26.562306 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.562288    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2pgcf"
Sep 08 10:02:26.572383 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.570769    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp"]
Sep 08 10:02:26.597725 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.597112    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8"]
Sep 08 10:02:26.602866 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:02:26.602813    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd18742f_fe5e_4fcb_b416_b566be8344e4.slice/crio-b3e4541ec09d7a44d88d4372f3ddfae7d643044e623744edc179aa245b935846 WatchSource:0}: Error finding container b3e4541ec09d7a44d88d4372f3ddfae7d643044e623744edc179aa245b935846: Status 404 returned error can't find the container with id b3e4541ec09d7a44d88d4372f3ddfae7d643044e623744edc179aa245b935846
Sep 08 10:02:26.619821 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.619791    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fnmlf"]
Sep 08 10:02:26.624519 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:02:26.624206    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacaccdd_dd0d_4bcd_a974_a065758240f7.slice/crio-a5d03688623597ed02ecea0c5555c995e1fea07280326a603354dfffdcff582c WatchSource:0}: Error finding container a5d03688623597ed02ecea0c5555c995e1fea07280326a603354dfffdcff582c: Status 404 returned error can't find the container with id a5d03688623597ed02ecea0c5555c995e1fea07280326a603354dfffdcff582c
Sep 08 10:02:26.850832 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.850754    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9h7fv"]
Sep 08 10:02:26.853562 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:02:26.853536    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81df3687_05a6_4b77_ac67_2f8c6626e906.slice/crio-40e21d9588984dd042871a1fdc4654419a782f9e08c0e614e7ebb99bf58fc562 WatchSource:0}: Error finding container 40e21d9588984dd042871a1fdc4654419a782f9e08c0e614e7ebb99bf58fc562: Status 404 returned error can't find the container with id 40e21d9588984dd042871a1fdc4654419a782f9e08c0e614e7ebb99bf58fc562
Sep 08 10:02:26.876231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.876163    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2pgcf"]
Sep 08 10:02:26.880897 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:02:26.880869    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a40235_67e9_4b28_ad0c_97751e2afe10.slice/crio-4c6fc9215935f8a027372aae9ca69fbadd334de6db6ac6cca20201217312c9ff WatchSource:0}: Error finding container 4c6fc9215935f8a027372aae9ca69fbadd334de6db6ac6cca20201217312c9ff: Status 404 returned error can't find the container with id 4c6fc9215935f8a027372aae9ca69fbadd334de6db6ac6cca20201217312c9ff
Sep 08 10:02:26.937382 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.937342    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8" event={"ID":"bd18742f-fe5e-4fcb-b416-b566be8344e4","Type":"ContainerStarted","Data":"b3e4541ec09d7a44d88d4372f3ddfae7d643044e623744edc179aa245b935846"}
Sep 08 10:02:26.939058 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.939011    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx" event={"ID":"13d91b33-2bf0-4bf4-b866-8b6bc251c280","Type":"ContainerStarted","Data":"65b412eda8cc94d0c2267ba716ab7dcb09b950d3a493190a3e9e5e0eef15179d"}
Sep 08 10:02:26.941064 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.941034    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5fbcc7c578-fzqlh" event={"ID":"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48","Type":"ContainerStarted","Data":"4ee61a80853556e0e060d915c1e366e12d3b18ea253de0f8c0229f76d4039852"}
Sep 08 10:02:26.941165 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.941086    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5fbcc7c578-fzqlh" event={"ID":"132efab2-1cc6-4fac-a9ef-61d9d8ab4c48","Type":"ContainerStarted","Data":"f6a170988ded7843e117b3a24f740744b085d34916af59203e7e0650a866c20d"}
Sep 08 10:02:26.943034 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.943010    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d946c986b-wvnk5" event={"ID":"7149cdff-fd81-49d1-9df7-9757dfd544d3","Type":"ContainerStarted","Data":"b784a1ca9ab02c2c1b81cbc6cef592af4aa0a94014d51ae54ff078a5d1046372"}
Sep 08 10:02:26.943214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.943054    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d946c986b-wvnk5" event={"ID":"7149cdff-fd81-49d1-9df7-9757dfd544d3","Type":"ContainerStarted","Data":"d3a99b9f105901f6a450b6e46d8030517b20d3053d1fd798a812dac6071190c8"}
Sep 08 10:02:26.943214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.943098    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:02:26.944848 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.944790    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp" event={"ID":"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464","Type":"ContainerStarted","Data":"aace436b3a3ba4ec930db2b52805e22ed3da23eddfa0b1a0636bcef405574efc"}
Sep 08 10:02:26.946300 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.946277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9h7fv" event={"ID":"81df3687-05a6-4b77-ac67-2f8c6626e906","Type":"ContainerStarted","Data":"40e21d9588984dd042871a1fdc4654419a782f9e08c0e614e7ebb99bf58fc562"}
Sep 08 10:02:26.947833 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.947808    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2pgcf" event={"ID":"a8a40235-67e9-4b28-ad0c-97751e2afe10","Type":"ContainerStarted","Data":"4c6fc9215935f8a027372aae9ca69fbadd334de6db6ac6cca20201217312c9ff"}
Sep 08 10:02:26.949497 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.949201    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fnmlf" event={"ID":"cacaccdd-dd0d-4bcd-a974-a065758240f7","Type":"ContainerStarted","Data":"a5d03688623597ed02ecea0c5555c995e1fea07280326a603354dfffdcff582c"}
Sep 08 10:02:26.963053 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.962950    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5fbcc7c578-fzqlh" podStartSLOduration=53.962932273 podStartE2EDuration="53.962932273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:26.959596081 +0000 UTC m=+66.800820488" watchObservedRunningTime="2026-09-08 10:02:26.962932273 +0000 UTC m=+66.804156657"
Sep 08 10:02:26.983037 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:26.982980    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d946c986b-wvnk5" podStartSLOduration=65.982963906 podStartE2EDuration="1m5.982963906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:26.981983332 +0000 UTC m=+66.823207759" watchObservedRunningTime="2026-09-08 10:02:26.982963906 +0000 UTC m=+66.824188289"
Sep 08 10:02:27.182816 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:27.182780    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:02:27.185730 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:27.185701    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:02:27.956317 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:27.956222    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvgzg" event={"ID":"d251d8ad-8bef-4a17-8cd3-7da32a35deb6","Type":"ContainerStarted","Data":"b8ac172609f512bb07250e9c980c17f37361adf05663582e65fc44f0b7ba312a"}
Sep 08 10:02:27.956728 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:27.956611    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:02:27.958058 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:27.958034    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5fbcc7c578-fzqlh"
Sep 08 10:02:27.976697 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:27.976646    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wvgzg" podStartSLOduration=16.869753541 podStartE2EDuration="23.976630614s" totalImagesPullingTime="7.106877073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:02:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:27.975413638 +0000 UTC m=+67.816638042" watchObservedRunningTime="2026-09-08 10:02:27.976630614 +0000 UTC m=+67.817854996"
Sep 08 10:02:28.961906 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:28.961875    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fnmlf" event={"ID":"cacaccdd-dd0d-4bcd-a974-a065758240f7","Type":"ContainerStarted","Data":"2454f0b8594ce384c46dc7e1ebe3a41a1d1a957686d4dddad58ba84eb78aa7db"}
Sep 08 10:02:28.962280 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:28.962077    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:02:28.962280 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:28.962129    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fnmlf" event={"ID":"cacaccdd-dd0d-4bcd-a974-a065758240f7","Type":"ContainerStarted","Data":"5a4ce874f906a883133ad018f83585405779a62bde43688407966dd6da0db149"}
Sep 08 10:02:28.986553 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:28.986500    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fnmlf" podStartSLOduration=33.40881312 podStartE2EDuration="34.986479265s" totalImagesPullingTime="1.577666145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:28.984907571 +0000 UTC m=+68.826131976" watchObservedRunningTime="2026-09-08 10:02:28.986479265 +0000 UTC m=+68.827703649"
Sep 08 10:02:29.969762 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:29.969726    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2pgcf" event={"ID":"a8a40235-67e9-4b28-ad0c-97751e2afe10","Type":"ContainerStarted","Data":"3f063b2977b0c00ab02fbd91bfec4adf0e1b62c780c527c3a805a4ff30eeaddc"}
Sep 08 10:02:29.970350 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:29.969788    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2pgcf" event={"ID":"a8a40235-67e9-4b28-ad0c-97751e2afe10","Type":"ContainerStarted","Data":"edfd52b6fe3dd4af46886c0af59fb315361be5b1853a5c1acc422eb8f70375f1"}
Sep 08 10:02:29.990766 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:29.990708    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2pgcf" podStartSLOduration=67.92448371 podStartE2EDuration="1m9.990686617s" totalImagesPullingTime="2.066202907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:29.988065561 +0000 UTC m=+69.829289945" watchObservedRunningTime="2026-09-08 10:02:29.990686617 +0000 UTC m=+69.831911001"
Sep 08 10:02:31.983444 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:31.983402    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8" event={"ID":"bd18742f-fe5e-4fcb-b416-b566be8344e4","Type":"ContainerStarted","Data":"16e4049e58e817b5359e4cb5edc13130097a60115c8ab627dbc08129668034ac"}
Sep 08 10:02:31.998833 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:31.998779    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wgqz8" podStartSLOduration=54.507851648 podStartE2EDuration="58.998761611s" totalImagesPullingTime="4.490909963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:31.99863642 +0000 UTC m=+71.839860805" watchObservedRunningTime="2026-09-08 10:02:31.998761611 +0000 UTC m=+71.839985997"
Sep 08 10:02:33.853553 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:33.853525    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pgp7f"
Sep 08 10:02:33.994857 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:33.994822    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp" event={"ID":"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464","Type":"ContainerStarted","Data":"9cde9969d5374cda791e052a685589a084a71db0535abe6f8e519cd08007c558"}
Sep 08 10:02:33.995038 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:33.994886    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp" event={"ID":"e0e0a0ff-1e4a-4f44-bfc1-0fb8e2b2e464","Type":"ContainerStarted","Data":"375f7cca0410fcd03428de4ae260bed10f36d0e80bc4687186521e5b7a114553"}
Sep 08 10:02:33.996944 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:33.996916    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9h7fv" event={"ID":"81df3687-05a6-4b77-ac67-2f8c6626e906","Type":"ContainerStarted","Data":"bc0cfd0375ae184f53660c0877f2d5736f5a343398b80e6c7648b8a190f88505"}
Sep 08 10:02:34.014277 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:34.014105    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-5p6mp" podStartSLOduration=54.025293753 podStartE2EDuration="1m1.014087045s" totalImagesPullingTime="6.988793292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:34.009352128 +0000 UTC m=+73.850576545" watchObservedRunningTime="2026-09-08 10:02:34.014087045 +0000 UTC m=+73.855311429"
Sep 08 10:02:34.026522 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:34.026483    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9h7fv" podStartSLOduration=66.250317092 podStartE2EDuration="1m13.026471599s" totalImagesPullingTime="6.776154507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:34.025419957 +0000 UTC m=+73.866644359" watchObservedRunningTime="2026-09-08 10:02:34.026471599 +0000 UTC m=+73.867695998"
Sep 08 10:02:35.002381 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:35.002290    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx" event={"ID":"13d91b33-2bf0-4bf4-b866-8b6bc251c280","Type":"ContainerStarted","Data":"76feb3c767740de7110de014652ca9d35c826cc511c8c1f89f8ffe23c1edf09b"}
Sep 08 10:02:35.019464 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:35.019415    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-7cwbx" podStartSLOduration=53.851968882 podStartE2EDuration="1m2.019400918s" totalImagesPullingTime="8.167432036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:35.018211842 +0000 UTC m=+74.859436228" watchObservedRunningTime="2026-09-08 10:02:35.019400918 +0000 UTC m=+74.860625301"
Sep 08 10:02:38.972088 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:38.972050    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fnmlf"
Sep 08 10:02:43.068124 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.068085    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5djbs"]
Sep 08 10:02:43.070546 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.070525    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.072830 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.072809    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 10:02:43.073621 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.073601    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f9gjt\""
Sep 08 10:02:43.073910 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.073892    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 10:02:43.073986 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.073909    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 10:02:43.074184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.074167    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 10:02:43.076765 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.076746    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"]
Sep 08 10:02:43.078932 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.078914    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.082262 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.082223    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cl2bb\""
Sep 08 10:02:43.082366 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.082282    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 08 10:02:43.082366 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.082297    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 08 10:02:43.082366 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.082311    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 08 10:02:43.097080 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.097060    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"]
Sep 08 10:02:43.161699 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.161675    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-wtmp\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.161830 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.161717    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-textfile\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.161830 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.161790    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-sys\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.161830 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.161818    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhbqn\" (UniqueName: \"kubernetes.io/projected/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-kube-api-access-nhbqn\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.161939 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.161856    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-metrics-client-ca\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.161939 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.161892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-root\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.161939 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.161917    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.162046 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.161992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-tls\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.162094 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.162064    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263009 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.262910    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-wtmp\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263009 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.262942    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac488f0b-dab6-4b4a-8227-de0f72e47555-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.263009 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.262968    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-textfile\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263009 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.262991    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-sys\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263026    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhbqn\" (UniqueName: \"kubernetes.io/projected/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-kube-api-access-nhbqn\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263042    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-sys\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263065    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-metrics-client-ca\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263096    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-root\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263116    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kprcp\" (UniqueName: \"kubernetes.io/projected/ac488f0b-dab6-4b4a-8227-de0f72e47555-kube-api-access-kprcp\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.263278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263093    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-wtmp\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263167    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac488f0b-dab6-4b4a-8227-de0f72e47555-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.263278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263196    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-tls\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263228    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac488f0b-dab6-4b4a-8227-de0f72e47555-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.263278 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263279    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263768 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263308    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ac488f0b-dab6-4b4a-8227-de0f72e47555-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.263768 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263342    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ac488f0b-dab6-4b4a-8227-de0f72e47555-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.263768 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263347    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-textfile\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263768 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:43.263356    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 08 10:02:43.263768 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263364    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-root\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263768 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:43.263449    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-tls podName:213ddfa0-a642-495a-a1f0-3038d0e2a5c4 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:43.763426137 +0000 UTC m=+83.604650516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-tls") pod "node-exporter-5djbs" (UID: "213ddfa0-a642-495a-a1f0-3038d0e2a5c4") : secret "node-exporter-tls" not found
Sep 08 10:02:43.263989 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263797    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-metrics-client-ca\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.263989 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.263806    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.265690 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.265668    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.271768 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.271748    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhbqn\" (UniqueName: \"kubernetes.io/projected/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-kube-api-access-nhbqn\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.364705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.364642    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ac488f0b-dab6-4b4a-8227-de0f72e47555-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.364705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.364685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac488f0b-dab6-4b4a-8227-de0f72e47555-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.364866 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.364735    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kprcp\" (UniqueName: \"kubernetes.io/projected/ac488f0b-dab6-4b4a-8227-de0f72e47555-kube-api-access-kprcp\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.364866 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.364761    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac488f0b-dab6-4b4a-8227-de0f72e47555-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.364866 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.364800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac488f0b-dab6-4b4a-8227-de0f72e47555-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.364866 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.364821    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ac488f0b-dab6-4b4a-8227-de0f72e47555-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.365192 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.365166    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ac488f0b-dab6-4b4a-8227-de0f72e47555-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.365489 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.365460    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ac488f0b-dab6-4b4a-8227-de0f72e47555-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.365580 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.365466    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac488f0b-dab6-4b4a-8227-de0f72e47555-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.367429 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.367406    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac488f0b-dab6-4b4a-8227-de0f72e47555-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.367517 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.367428    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac488f0b-dab6-4b4a-8227-de0f72e47555-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.372555 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.372534    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kprcp\" (UniqueName: \"kubernetes.io/projected/ac488f0b-dab6-4b4a-8227-de0f72e47555-kube-api-access-kprcp\") pod \"kube-state-metrics-74cb57df7-xvrdg\" (UID: \"ac488f0b-dab6-4b4a-8227-de0f72e47555\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.387415 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.387397    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"
Sep 08 10:02:43.657920 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.657897    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg"]
Sep 08 10:02:43.659606 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:02:43.659574    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac488f0b_dab6_4b4a_8227_de0f72e47555.slice/crio-3596d396ea3d75d43df8656454720a1b0f1266595e078a9ba7bc7b61d158e3e2 WatchSource:0}: Error finding container 3596d396ea3d75d43df8656454720a1b0f1266595e078a9ba7bc7b61d158e3e2: Status 404 returned error can't find the container with id 3596d396ea3d75d43df8656454720a1b0f1266595e078a9ba7bc7b61d158e3e2
Sep 08 10:02:43.768751 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.768725    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-tls\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.770859 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.770835    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/213ddfa0-a642-495a-a1f0-3038d0e2a5c4-node-exporter-tls\") pod \"node-exporter-5djbs\" (UID: \"213ddfa0-a642-495a-a1f0-3038d0e2a5c4\") " pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.980211 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:43.980131    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5djbs"
Sep 08 10:02:43.991189 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:02:43.991158    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213ddfa0_a642_495a_a1f0_3038d0e2a5c4.slice/crio-4d3e602653382085181f0dc8c13e636bf7ef33b32038783ef861efc14c6815b7 WatchSource:0}: Error finding container 4d3e602653382085181f0dc8c13e636bf7ef33b32038783ef861efc14c6815b7: Status 404 returned error can't find the container with id 4d3e602653382085181f0dc8c13e636bf7ef33b32038783ef861efc14c6815b7
Sep 08 10:02:44.037060 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:44.037022    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5djbs" event={"ID":"213ddfa0-a642-495a-a1f0-3038d0e2a5c4","Type":"ContainerStarted","Data":"4d3e602653382085181f0dc8c13e636bf7ef33b32038783ef861efc14c6815b7"}
Sep 08 10:02:44.038202 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:44.038181    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg" event={"ID":"ac488f0b-dab6-4b4a-8227-de0f72e47555","Type":"ContainerStarted","Data":"3596d396ea3d75d43df8656454720a1b0f1266595e078a9ba7bc7b61d158e3e2"}
Sep 08 10:02:46.053436 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:46.053400    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg" event={"ID":"ac488f0b-dab6-4b4a-8227-de0f72e47555","Type":"ContainerStarted","Data":"e70d0653f9caf2fe824b52942ae4442c84968160daeb29b6f67c26862e4a3251"}
Sep 08 10:02:46.053842 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:46.053499    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg" event={"ID":"ac488f0b-dab6-4b4a-8227-de0f72e47555","Type":"ContainerStarted","Data":"20da0bfc76fd7a22818a5c7c2e37dd5c2702ed80e4b4a09a6ca4cb868a07bd62"}
Sep 08 10:02:46.053842 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:46.053521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg" event={"ID":"ac488f0b-dab6-4b4a-8227-de0f72e47555","Type":"ContainerStarted","Data":"d8f8ee5858bfb7c7546e4478a2c9b22a1ecb6be10c6fadd1fdf46ea532d6963b"}
Sep 08 10:02:46.096929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:46.096877    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-xvrdg" podStartSLOduration=1.382600733 podStartE2EDuration="3.096861407s" totalImagesPullingTime="1.714260674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:46.096115711 +0000 UTC m=+85.937340093" watchObservedRunningTime="2026-09-08 10:02:46.096861407 +0000 UTC m=+85.938085789"
Sep 08 10:02:47.833873 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:47.833842    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt"]
Sep 08 10:02:47.836733 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:47.836715    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt"
Sep 08 10:02:47.839154 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:47.839137    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 08 10:02:47.839273 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:47.839225    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-87ftw\""
Sep 08 10:02:47.844623 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:47.844600    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt"]
Sep 08 10:02:47.960874 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:47.960850    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:02:48.003465 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.003431    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktvp\" (UniqueName: \"kubernetes.io/projected/0190d889-1bbf-4260-a6b7-275de53ebbd8-kube-api-access-7ktvp\") pod \"monitoring-plugin-85bbbbbf4-trzkt\" (UID: \"0190d889-1bbf-4260-a6b7-275de53ebbd8\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt"
Sep 08 10:02:48.003607 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.003473    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0190d889-1bbf-4260-a6b7-275de53ebbd8-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-trzkt\" (UID: \"0190d889-1bbf-4260-a6b7-275de53ebbd8\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt"
Sep 08 10:02:48.103975 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.103885    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktvp\" (UniqueName: \"kubernetes.io/projected/0190d889-1bbf-4260-a6b7-275de53ebbd8-kube-api-access-7ktvp\") pod \"monitoring-plugin-85bbbbbf4-trzkt\" (UID: \"0190d889-1bbf-4260-a6b7-275de53ebbd8\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt"
Sep 08 10:02:48.103975 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.103943    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0190d889-1bbf-4260-a6b7-275de53ebbd8-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-trzkt\" (UID: \"0190d889-1bbf-4260-a6b7-275de53ebbd8\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt"
Sep 08 10:02:48.104235 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:48.104096    2559 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 08 10:02:48.104235 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:48.104177    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0190d889-1bbf-4260-a6b7-275de53ebbd8-monitoring-plugin-cert podName:0190d889-1bbf-4260-a6b7-275de53ebbd8 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:48.604156728 +0000 UTC m=+88.445381118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0190d889-1bbf-4260-a6b7-275de53ebbd8-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-trzkt" (UID: "0190d889-1bbf-4260-a6b7-275de53ebbd8") : secret "monitoring-plugin-cert" not found
Sep 08 10:02:48.112509 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.112485    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktvp\" (UniqueName: \"kubernetes.io/projected/0190d889-1bbf-4260-a6b7-275de53ebbd8-kube-api-access-7ktvp\") pod \"monitoring-plugin-85bbbbbf4-trzkt\" (UID: \"0190d889-1bbf-4260-a6b7-275de53ebbd8\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt"
Sep 08 10:02:48.278527 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.278493    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"]
Sep 08 10:02:48.280936 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.280918    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.283198 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.283179    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 08 10:02:48.283419 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.283401    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 08 10:02:48.283646 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.283583    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 08 10:02:48.283646 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.283620    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 08 10:02:48.283646 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.283633    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 08 10:02:48.283973 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.283958    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vkk97\""
Sep 08 10:02:48.289227 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.289193    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 08 10:02:48.294849 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.294831    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"]
Sep 08 10:02:48.406756 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.406725    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eead0fa1-afe9-4343-979e-bf006a2dce3c-federate-client-tls\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.406873 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.406780    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eead0fa1-afe9-4343-979e-bf006a2dce3c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.406873 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.406812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eead0fa1-afe9-4343-979e-bf006a2dce3c-metrics-client-ca\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.406873 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.406838    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xhdb\" (UniqueName: \"kubernetes.io/projected/eead0fa1-afe9-4343-979e-bf006a2dce3c-kube-api-access-6xhdb\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.406873 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.406872    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eead0fa1-afe9-4343-979e-bf006a2dce3c-serving-certs-ca-bundle\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.407088 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.406895    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eead0fa1-afe9-4343-979e-bf006a2dce3c-telemeter-client-tls\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.407088 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.407008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eead0fa1-afe9-4343-979e-bf006a2dce3c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.407088 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.407056    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eead0fa1-afe9-4343-979e-bf006a2dce3c-secret-telemeter-client\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.508413 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.508382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eead0fa1-afe9-4343-979e-bf006a2dce3c-federate-client-tls\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.508588 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.508451    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eead0fa1-afe9-4343-979e-bf006a2dce3c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.508588 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.508496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eead0fa1-afe9-4343-979e-bf006a2dce3c-metrics-client-ca\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.508588 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.508525    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhdb\" (UniqueName: \"kubernetes.io/projected/eead0fa1-afe9-4343-979e-bf006a2dce3c-kube-api-access-6xhdb\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.508588 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.508568    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eead0fa1-afe9-4343-979e-bf006a2dce3c-serving-certs-ca-bundle\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.508818 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.508596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eead0fa1-afe9-4343-979e-bf006a2dce3c-telemeter-client-tls\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.508818 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.508664    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eead0fa1-afe9-4343-979e-bf006a2dce3c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.508818 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.508701    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eead0fa1-afe9-4343-979e-bf006a2dce3c-secret-telemeter-client\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.509444 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.509416    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eead0fa1-afe9-4343-979e-bf006a2dce3c-metrics-client-ca\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.509617 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.509460    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eead0fa1-afe9-4343-979e-bf006a2dce3c-serving-certs-ca-bundle\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.509869 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.509841    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eead0fa1-afe9-4343-979e-bf006a2dce3c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.511585 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.511390    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eead0fa1-afe9-4343-979e-bf006a2dce3c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.511585 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.511544    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eead0fa1-afe9-4343-979e-bf006a2dce3c-federate-client-tls\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.511771 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.511750    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eead0fa1-afe9-4343-979e-bf006a2dce3c-telemeter-client-tls\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.517013 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.512187    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eead0fa1-afe9-4343-979e-bf006a2dce3c-secret-telemeter-client\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.518450 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.518425    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xhdb\" (UniqueName: \"kubernetes.io/projected/eead0fa1-afe9-4343-979e-bf006a2dce3c-kube-api-access-6xhdb\") pod \"telemeter-client-8ff4d8c67-bb5t9\" (UID: \"eead0fa1-afe9-4343-979e-bf006a2dce3c\") " pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.591453 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.591429    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"
Sep 08 10:02:48.609417 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.609395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0190d889-1bbf-4260-a6b7-275de53ebbd8-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-trzkt\" (UID: \"0190d889-1bbf-4260-a6b7-275de53ebbd8\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt"
Sep 08 10:02:48.611721 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.611698    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0190d889-1bbf-4260-a6b7-275de53ebbd8-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-trzkt\" (UID: \"0190d889-1bbf-4260-a6b7-275de53ebbd8\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt"
Sep 08 10:02:48.745881 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.745803    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt"
Sep 08 10:02:48.873544 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:48.873517    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9"]
Sep 08 10:02:48.875414 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:02:48.875379    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeead0fa1_afe9_4343_979e_bf006a2dce3c.slice/crio-6b4c727e7b865668dd20e23a5bf3025f109f7125ecbfbb0d91d18abe76ca0981 WatchSource:0}: Error finding container 6b4c727e7b865668dd20e23a5bf3025f109f7125ecbfbb0d91d18abe76ca0981: Status 404 returned error can't find the container with id 6b4c727e7b865668dd20e23a5bf3025f109f7125ecbfbb0d91d18abe76ca0981
Sep 08 10:02:49.013941 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.013804    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt"]
Sep 08 10:02:49.016368 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:02:49.016346    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0190d889_1bbf_4260_a6b7_275de53ebbd8.slice/crio-17794b104b68fd2ca7e83ab86d49a355e564fc5045bf314b48b48d19f7679737 WatchSource:0}: Error finding container 17794b104b68fd2ca7e83ab86d49a355e564fc5045bf314b48b48d19f7679737: Status 404 returned error can't find the container with id 17794b104b68fd2ca7e83ab86d49a355e564fc5045bf314b48b48d19f7679737
Sep 08 10:02:49.070296 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.070261    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9" event={"ID":"eead0fa1-afe9-4343-979e-bf006a2dce3c","Type":"ContainerStarted","Data":"6b4c727e7b865668dd20e23a5bf3025f109f7125ecbfbb0d91d18abe76ca0981"}
Sep 08 10:02:49.071421 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.071389    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt" event={"ID":"0190d889-1bbf-4260-a6b7-275de53ebbd8","Type":"ContainerStarted","Data":"17794b104b68fd2ca7e83ab86d49a355e564fc5045bf314b48b48d19f7679737"}
Sep 08 10:02:49.298481 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.298388    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 10:02:49.302100 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.302068    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.304808 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.304781    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 10:02:49.305284 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.305145    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 10:02:49.305284 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.305206    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5tsftlvnltlmv\""
Sep 08 10:02:49.305284 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.305226    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 10:02:49.305284 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.305226    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 10:02:49.305563 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.305474    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 10:02:49.305563 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.305486    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 10:02:49.305664 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.305603    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 10:02:49.305747 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.305726    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 10:02:49.305812 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.305740    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 10:02:49.305812 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.305733    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 10:02:49.306084 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.306065    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 10:02:49.306168 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.306107    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 10:02:49.306168 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.306127    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 10:02:49.306168 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.306130    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8vgft\""
Sep 08 10:02:49.306340 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.306159    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 10:02:49.308504 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.308482    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 10:02:49.314424 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314517 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314517 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314465    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314593 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314530    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314624 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314590    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314654 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314628    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314683 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314653    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314683 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314667    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314752 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314691    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314752 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314742    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314809 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314765    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314809 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314780    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314870 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-config\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314870 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314826    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce127e3c-6c5b-413c-b506-fc36dac79ae2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314870 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314870 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314864    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce127e3c-6c5b-413c-b506-fc36dac79ae2-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314988 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314988 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314923    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314988 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314953    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.314988 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.314968    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stc58\" (UniqueName: \"kubernetes.io/projected/ce127e3c-6c5b-413c-b506-fc36dac79ae2-kube-api-access-stc58\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.321219 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.321196    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 10:02:49.416160 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416116    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416171    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416192    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416207    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416231    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416279    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416313    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416337    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-config\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416383    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce127e3c-6c5b-413c-b506-fc36dac79ae2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416406    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416428    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce127e3c-6c5b-413c-b506-fc36dac79ae2-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416448    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416506    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416536    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416560    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stc58\" (UniqueName: \"kubernetes.io/projected/ce127e3c-6c5b-413c-b506-fc36dac79ae2-kube-api-access-stc58\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416593    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416625    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.416705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.417331 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.416919    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.417493 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.417465    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.418101 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.417759    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.419632 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.419317    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.419752 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.419719    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.419837 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.419809    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce127e3c-6c5b-413c-b506-fc36dac79ae2-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.419952 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.419880    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.420687 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.420354    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.420687 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.420396    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.420687 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:02:49.420536    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-trusted-ca-bundle podName:ce127e3c-6c5b-413c-b506-fc36dac79ae2 nodeName:}" failed. No retries permitted until 2026-09-08 10:02:49.920515528 +0000 UTC m=+89.761739895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-trusted-ca-bundle") pod "prometheus-k8s-0" (UID: "ce127e3c-6c5b-413c-b506-fc36dac79ae2") : configmap references non-existent config key: ca-bundle.crt
Sep 08 10:02:49.421032 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.421007    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.421509 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.421458    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.421608 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.421590    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.421695 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.421619    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.422626 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.422600    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.422954 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.422933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce127e3c-6c5b-413c-b506-fc36dac79ae2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.423503 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.423439    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-config\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.423990 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.423950    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce127e3c-6c5b-413c-b506-fc36dac79ae2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.424081 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.423986    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.430780 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.430759    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stc58\" (UniqueName: \"kubernetes.io/projected/ce127e3c-6c5b-413c-b506-fc36dac79ae2-kube-api-access-stc58\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.922114 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.922078    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:49.923105 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:49.923056    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce127e3c-6c5b-413c-b506-fc36dac79ae2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce127e3c-6c5b-413c-b506-fc36dac79ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:50.080661 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:50.080623    2559 generic.go:356] "Generic (PLEG): container finished" podID="213ddfa0-a642-495a-a1f0-3038d0e2a5c4" containerID="7a2c20392e17d63ad971bf49fa4f3f33519f6ea317e34bdbb14b345bf448c40a" exitCode=0
Sep 08 10:02:50.080848 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:50.080686    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5djbs" event={"ID":"213ddfa0-a642-495a-a1f0-3038d0e2a5c4","Type":"ContainerDied","Data":"7a2c20392e17d63ad971bf49fa4f3f33519f6ea317e34bdbb14b345bf448c40a"}
Sep 08 10:02:50.213066 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:50.212996    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:02:51.087002 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:51.086970    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5djbs" event={"ID":"213ddfa0-a642-495a-a1f0-3038d0e2a5c4","Type":"ContainerStarted","Data":"e244e8767c5eff87ab326bcd2e19c2339e97204c058f21d5470a1ddf39340af4"}
Sep 08 10:02:51.087476 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:51.087075    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5djbs" event={"ID":"213ddfa0-a642-495a-a1f0-3038d0e2a5c4","Type":"ContainerStarted","Data":"40b7a808c28682164c2a7868d872bae16acb81269862a01bedf622665875321f"}
Sep 08 10:02:51.088598 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:51.088570    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt" event={"ID":"0190d889-1bbf-4260-a6b7-275de53ebbd8","Type":"ContainerStarted","Data":"8bbb959927f2b53c13b6af36ec86635d7b80405298ff214598a300ed64db4889"}
Sep 08 10:02:51.088726 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:51.088701    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt"
Sep 08 10:02:51.090206 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:51.090185    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9" event={"ID":"eead0fa1-afe9-4343-979e-bf006a2dce3c","Type":"ContainerStarted","Data":"ba85163fd668634afc91e61133181cc1a04396ee145669c21a24b3929b7096fc"}
Sep 08 10:02:51.094185 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:51.094166    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt"
Sep 08 10:02:51.106873 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:51.106834    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5djbs" podStartSLOduration=2.473769944 podStartE2EDuration="8.106821295s" totalImagesPullingTime="5.558014994s" totalInitContainerRuntime="75.036357ms" isStatefulPod=true podCreationTimestamp="2026-09-08 10:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:51.104792685 +0000 UTC m=+90.946017106" watchObservedRunningTime="2026-09-08 10:02:51.106821295 +0000 UTC m=+90.948045714"
Sep 08 10:02:51.119585 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:51.119545    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-trzkt" podStartSLOduration=2.178135778 podStartE2EDuration="4.119531112s" totalImagesPullingTime="1.941395334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:51.118143332 +0000 UTC m=+90.959367736" watchObservedRunningTime="2026-09-08 10:02:51.119531112 +0000 UTC m=+90.960755565"
Sep 08 10:02:51.231266 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:51.231229    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 10:02:51.235212 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:02:51.235181    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce127e3c_6c5b_413c_b506_fc36dac79ae2.slice/crio-beab31380877fa452a3ac6ba8efad11ff40ae251e645c509463caa6018f2a944 WatchSource:0}: Error finding container beab31380877fa452a3ac6ba8efad11ff40ae251e645c509463caa6018f2a944: Status 404 returned error can't find the container with id beab31380877fa452a3ac6ba8efad11ff40ae251e645c509463caa6018f2a944
Sep 08 10:02:52.096597 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:52.096568    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9" event={"ID":"eead0fa1-afe9-4343-979e-bf006a2dce3c","Type":"ContainerStarted","Data":"9f8abfcc19075841b11b0b4432e588fb51630d101364f549cc0ea5634e2f6de2"}
Sep 08 10:02:52.096948 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:52.096624    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9" event={"ID":"eead0fa1-afe9-4343-979e-bf006a2dce3c","Type":"ContainerStarted","Data":"35404ab0e901224072ce4066e2c060cbc3f955e5ba76fb796f71303e4107224c"}
Sep 08 10:02:52.097749 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:52.097728    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce127e3c-6c5b-413c-b506-fc36dac79ae2","Type":"ContainerStarted","Data":"beab31380877fa452a3ac6ba8efad11ff40ae251e645c509463caa6018f2a944"}
Sep 08 10:02:52.117760 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:52.117630    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8ff4d8c67-bb5t9" podStartSLOduration=1.119973816 podStartE2EDuration="4.117613402s" totalImagesPullingTime="2.997639586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:02:52.116296004 +0000 UTC m=+91.957520406" watchObservedRunningTime="2026-09-08 10:02:52.117613402 +0000 UTC m=+91.958837786"
Sep 08 10:02:53.007211 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:53.007178    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d946c986b-wvnk5"]
Sep 08 10:02:55.112379 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:55.112337    2559 generic.go:356] "Generic (PLEG): container finished" podID="ce127e3c-6c5b-413c-b506-fc36dac79ae2" containerID="f472a762b4256c42999e6fc0c2e7cede82ea654a105d99af9c9b381d03a6c755" exitCode=0
Sep 08 10:02:55.112847 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:02:55.112443    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce127e3c-6c5b-413c-b506-fc36dac79ae2","Type":"ContainerDied","Data":"f472a762b4256c42999e6fc0c2e7cede82ea654a105d99af9c9b381d03a6c755"}
Sep 08 10:03:01.140359 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:01.140322    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce127e3c-6c5b-413c-b506-fc36dac79ae2","Type":"ContainerStarted","Data":"d6d56daee2cef378283d4037b447da113808277d821c2569270ddcee5af091eb"}
Sep 08 10:03:01.140737 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:01.140390    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce127e3c-6c5b-413c-b506-fc36dac79ae2","Type":"ContainerStarted","Data":"d62bf7b210827b73fb2f1c7467329c01a21b1dbbfe75f42b4a8233df4a2e2530"}
Sep 08 10:03:04.157147 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:04.157105    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce127e3c-6c5b-413c-b506-fc36dac79ae2","Type":"ContainerStarted","Data":"83145bdb6f51125b64540a2d85cddccd0ee0127b64e713aa86aa2ba8722e9c9e"}
Sep 08 10:03:04.157533 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:04.157183    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce127e3c-6c5b-413c-b506-fc36dac79ae2","Type":"ContainerStarted","Data":"6687f0023b4a9d3712f5f45b876c30d65ee06d4c763fe1db884c0869e75ff14c"}
Sep 08 10:03:04.157533 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:04.157221    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce127e3c-6c5b-413c-b506-fc36dac79ae2","Type":"ContainerStarted","Data":"e81e46a50636ce999fd540fbfaca1338a8ad311df6e8df61e79d7c07581ae354"}
Sep 08 10:03:04.157533 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:04.157278    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce127e3c-6c5b-413c-b506-fc36dac79ae2","Type":"ContainerStarted","Data":"8f5ad4ec92f50c132d613b4896dcc84e7b19a6c154f79f12933fb0ab17efd729"}
Sep 08 10:03:04.199769 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:04.199720    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.650840178 podStartE2EDuration="15.199703764s" totalImagesPullingTime="11.531636928s" totalInitContainerRuntime="17.226658ms" isStatefulPod=false podCreationTimestamp="2026-09-08 10:02:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:03:04.197573807 +0000 UTC m=+104.038798207" watchObservedRunningTime="2026-09-08 10:03:04.199703764 +0000 UTC m=+104.040928160"
Sep 08 10:03:05.213321 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:05.213288    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:03:18.026867 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.026809    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d946c986b-wvnk5" podUID="7149cdff-fd81-49d1-9df7-9757dfd544d3" containerName="registry" containerID="cri-o://b784a1ca9ab02c2c1b81cbc6cef592af4aa0a94014d51ae54ff078a5d1046372" gracePeriod=30
Sep 08 10:03:18.218077 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.218034    2559 generic.go:356] "Generic (PLEG): container finished" podID="7149cdff-fd81-49d1-9df7-9757dfd544d3" containerID="b784a1ca9ab02c2c1b81cbc6cef592af4aa0a94014d51ae54ff078a5d1046372" exitCode=0
Sep 08 10:03:18.218234 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.218099    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d946c986b-wvnk5" event={"ID":"7149cdff-fd81-49d1-9df7-9757dfd544d3","Type":"ContainerDied","Data":"b784a1ca9ab02c2c1b81cbc6cef592af4aa0a94014d51ae54ff078a5d1046372"}
Sep 08 10:03:18.262413 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.262388    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:03:18.382059 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.382001    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7149cdff-fd81-49d1-9df7-9757dfd544d3-trusted-ca\") pod \"7149cdff-fd81-49d1-9df7-9757dfd544d3\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") "
Sep 08 10:03:18.382059 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.382037    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7149cdff-fd81-49d1-9df7-9757dfd544d3-ca-trust-extracted\") pod \"7149cdff-fd81-49d1-9df7-9757dfd544d3\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") "
Sep 08 10:03:18.382059 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.382056    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7149cdff-fd81-49d1-9df7-9757dfd544d3-installation-pull-secrets\") pod \"7149cdff-fd81-49d1-9df7-9757dfd544d3\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") "
Sep 08 10:03:18.382315 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.382079    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwrwt\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-kube-api-access-zwrwt\") pod \"7149cdff-fd81-49d1-9df7-9757dfd544d3\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") "
Sep 08 10:03:18.382315 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.382155    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-certificates\") pod \"7149cdff-fd81-49d1-9df7-9757dfd544d3\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") "
Sep 08 10:03:18.382315 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.382218    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls\") pod \"7149cdff-fd81-49d1-9df7-9757dfd544d3\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") "
Sep 08 10:03:18.382315 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.382272    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7149cdff-fd81-49d1-9df7-9757dfd544d3-image-registry-private-configuration\") pod \"7149cdff-fd81-49d1-9df7-9757dfd544d3\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") "
Sep 08 10:03:18.382315 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.382302    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-bound-sa-token\") pod \"7149cdff-fd81-49d1-9df7-9757dfd544d3\" (UID: \"7149cdff-fd81-49d1-9df7-9757dfd544d3\") "
Sep 08 10:03:18.382568 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.382482    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7149cdff-fd81-49d1-9df7-9757dfd544d3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7149cdff-fd81-49d1-9df7-9757dfd544d3" (UID: "7149cdff-fd81-49d1-9df7-9757dfd544d3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:03:18.382663 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.382642    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7149cdff-fd81-49d1-9df7-9757dfd544d3-trusted-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:03:18.383289 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.383234    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7149cdff-fd81-49d1-9df7-9757dfd544d3" (UID: "7149cdff-fd81-49d1-9df7-9757dfd544d3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:03:18.385027 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.384996    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7149cdff-fd81-49d1-9df7-9757dfd544d3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7149cdff-fd81-49d1-9df7-9757dfd544d3" (UID: "7149cdff-fd81-49d1-9df7-9757dfd544d3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:03:18.385126 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.385037    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7149cdff-fd81-49d1-9df7-9757dfd544d3" (UID: "7149cdff-fd81-49d1-9df7-9757dfd544d3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:03:18.385126 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.385087    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7149cdff-fd81-49d1-9df7-9757dfd544d3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7149cdff-fd81-49d1-9df7-9757dfd544d3" (UID: "7149cdff-fd81-49d1-9df7-9757dfd544d3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:03:18.385126 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.385113    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-kube-api-access-zwrwt" (OuterVolumeSpecName: "kube-api-access-zwrwt") pod "7149cdff-fd81-49d1-9df7-9757dfd544d3" (UID: "7149cdff-fd81-49d1-9df7-9757dfd544d3"). InnerVolumeSpecName "kube-api-access-zwrwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:03:18.385258 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.385192    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7149cdff-fd81-49d1-9df7-9757dfd544d3" (UID: "7149cdff-fd81-49d1-9df7-9757dfd544d3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:03:18.390593 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.390567    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7149cdff-fd81-49d1-9df7-9757dfd544d3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7149cdff-fd81-49d1-9df7-9757dfd544d3" (UID: "7149cdff-fd81-49d1-9df7-9757dfd544d3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:03:18.483231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.483208    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-certificates\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:03:18.483231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.483230    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-registry-tls\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:03:18.483361 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.483259    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7149cdff-fd81-49d1-9df7-9757dfd544d3-image-registry-private-configuration\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:03:18.483361 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.483285    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-bound-sa-token\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:03:18.483361 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.483300    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7149cdff-fd81-49d1-9df7-9757dfd544d3-ca-trust-extracted\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:03:18.483361 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.483307    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7149cdff-fd81-49d1-9df7-9757dfd544d3-installation-pull-secrets\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:03:18.483361 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:18.483316    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwrwt\" (UniqueName: \"kubernetes.io/projected/7149cdff-fd81-49d1-9df7-9757dfd544d3-kube-api-access-zwrwt\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:03:19.223739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:19.223709    2559 generic.go:356] "Generic (PLEG): container finished" podID="1ba9dc85-d817-4dde-bb41-b50a8fd3d544" containerID="96f07faf674b2c1389ed0919079aeb0e8a9c206c54348fe6921675fc3cc5f37f" exitCode=0
Sep 08 10:03:19.224140 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:19.223788    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf" event={"ID":"1ba9dc85-d817-4dde-bb41-b50a8fd3d544","Type":"ContainerDied","Data":"96f07faf674b2c1389ed0919079aeb0e8a9c206c54348fe6921675fc3cc5f37f"}
Sep 08 10:03:19.224187 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:19.224178    2559 scope.go:117] "RemoveContainer" containerID="96f07faf674b2c1389ed0919079aeb0e8a9c206c54348fe6921675fc3cc5f37f"
Sep 08 10:03:19.225203 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:19.225180    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d946c986b-wvnk5" event={"ID":"7149cdff-fd81-49d1-9df7-9757dfd544d3","Type":"ContainerDied","Data":"d3a99b9f105901f6a450b6e46d8030517b20d3053d1fd798a812dac6071190c8"}
Sep 08 10:03:19.225331 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:19.225209    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d946c986b-wvnk5"
Sep 08 10:03:19.225331 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:19.225225    2559 scope.go:117] "RemoveContainer" containerID="b784a1ca9ab02c2c1b81cbc6cef592af4aa0a94014d51ae54ff078a5d1046372"
Sep 08 10:03:19.258640 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:19.258616    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d946c986b-wvnk5"]
Sep 08 10:03:19.265334 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:19.265176    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d946c986b-wvnk5"]
Sep 08 10:03:20.231994 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:20.231962    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-dwmlf" event={"ID":"1ba9dc85-d817-4dde-bb41-b50a8fd3d544","Type":"ContainerStarted","Data":"5d3598e50a0c18e6eb5d2a34555665d67e24d207f8be08a69fe9233264077c89"}
Sep 08 10:03:20.636942 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:20.636910    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7149cdff-fd81-49d1-9df7-9757dfd544d3" path="/var/lib/kubelet/pods/7149cdff-fd81-49d1-9df7-9757dfd544d3/volumes"
Sep 08 10:03:28.267486 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:28.267459    2559 generic.go:356] "Generic (PLEG): container finished" podID="633deb4b-3d4a-47c5-b6ef-a8d8a803c521" containerID="c3a604a358912c34c4bef492bc3accab9a48ab7a8cebf9a3ffa5bc555303181b" exitCode=0
Sep 08 10:03:28.267799 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:28.267533    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c" event={"ID":"633deb4b-3d4a-47c5-b6ef-a8d8a803c521","Type":"ContainerDied","Data":"c3a604a358912c34c4bef492bc3accab9a48ab7a8cebf9a3ffa5bc555303181b"}
Sep 08 10:03:28.267843 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:28.267834    2559 scope.go:117] "RemoveContainer" containerID="c3a604a358912c34c4bef492bc3accab9a48ab7a8cebf9a3ffa5bc555303181b"
Sep 08 10:03:29.273869 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:29.273836    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p9j4c" event={"ID":"633deb4b-3d4a-47c5-b6ef-a8d8a803c521","Type":"ContainerStarted","Data":"a4e523ef72d4346a1030db9b7788f38bbd3ce5d1c2a4d078701929f7900bb5b1"}
Sep 08 10:03:32.288365 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:32.288332    2559 generic.go:356] "Generic (PLEG): container finished" podID="5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9" containerID="5f5833125410b72aa65f21863df6b408d943b8d2c2a5458827c9d87a19252365" exitCode=0
Sep 08 10:03:32.288818 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:32.288405    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q" event={"ID":"5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9","Type":"ContainerDied","Data":"5f5833125410b72aa65f21863df6b408d943b8d2c2a5458827c9d87a19252365"}
Sep 08 10:03:32.288818 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:32.288730    2559 scope.go:117] "RemoveContainer" containerID="5f5833125410b72aa65f21863df6b408d943b8d2c2a5458827c9d87a19252365"
Sep 08 10:03:33.296451 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:33.296421    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ls69q" event={"ID":"5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9","Type":"ContainerStarted","Data":"e884548352256a633da75851482bbe66920a35a32c65de14b5f3494792cd266d"}
Sep 08 10:03:50.213289 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:50.213252    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:03:50.227692 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:50.227664    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:03:50.378860 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:03:50.378833    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 10:06:20.577058 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:20.577025    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:06:20.582460 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:20.582425    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:06:20.589687 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:20.589665    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 10:06:43.594745 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.594707    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"]
Sep 08 10:06:43.596204 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.595184    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7149cdff-fd81-49d1-9df7-9757dfd544d3" containerName="registry"
Sep 08 10:06:43.596204 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.595200    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7149cdff-fd81-49d1-9df7-9757dfd544d3" containerName="registry"
Sep 08 10:06:43.596204 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.595292    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7149cdff-fd81-49d1-9df7-9757dfd544d3" containerName="registry"
Sep 08 10:06:43.597233 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.597216    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.599480 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.599457    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 10:06:43.599568 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.599468    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 10:06:43.599568 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.599545    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 10:06:43.600038 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.600023    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nb2g9\""
Sep 08 10:06:43.600162 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.600139    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 10:06:43.611906 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.611886    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"]
Sep 08 10:06:43.725319 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.725290    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbprm\" (UniqueName: \"kubernetes.io/projected/707073bb-7800-4be5-93d5-87d02b168c1e-kube-api-access-qbprm\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.725465 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.725325    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/707073bb-7800-4be5-93d5-87d02b168c1e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.725465 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.725359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/707073bb-7800-4be5-93d5-87d02b168c1e-webhook-cert\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.725465 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.725442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/707073bb-7800-4be5-93d5-87d02b168c1e-manifests\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.725465 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.725461    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/707073bb-7800-4be5-93d5-87d02b168c1e-tmp\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.825945 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.825920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/707073bb-7800-4be5-93d5-87d02b168c1e-manifests\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.826064 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.825955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/707073bb-7800-4be5-93d5-87d02b168c1e-tmp\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.826064 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.826008    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbprm\" (UniqueName: \"kubernetes.io/projected/707073bb-7800-4be5-93d5-87d02b168c1e-kube-api-access-qbprm\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.826064 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.826038    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/707073bb-7800-4be5-93d5-87d02b168c1e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.826222 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.826163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/707073bb-7800-4be5-93d5-87d02b168c1e-webhook-cert\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.826340 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.826317    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/707073bb-7800-4be5-93d5-87d02b168c1e-manifests\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.828398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.828369    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/707073bb-7800-4be5-93d5-87d02b168c1e-tmp\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.828568 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.828546    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/707073bb-7800-4be5-93d5-87d02b168c1e-webhook-cert\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.828612 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.828565    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/707073bb-7800-4be5-93d5-87d02b168c1e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.833531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.833506    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbprm\" (UniqueName: \"kubernetes.io/projected/707073bb-7800-4be5-93d5-87d02b168c1e-kube-api-access-qbprm\") pod \"opendatahub-operator-controller-manager-56b6d9c7b-hvsvs\" (UID: \"707073bb-7800-4be5-93d5-87d02b168c1e\") " pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:43.907239 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:43.907214    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:44.189121 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:44.189098    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"]
Sep 08 10:06:44.191600 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:06:44.191576    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707073bb_7800_4be5_93d5_87d02b168c1e.slice/crio-b19a17ac52c4e1eb29a07a332870c254a277c82a955f0f4ff0f86692da33c792 WatchSource:0}: Error finding container b19a17ac52c4e1eb29a07a332870c254a277c82a955f0f4ff0f86692da33c792: Status 404 returned error can't find the container with id b19a17ac52c4e1eb29a07a332870c254a277c82a955f0f4ff0f86692da33c792
Sep 08 10:06:44.193314 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:44.193296    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 10:06:45.063164 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:45.063126    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs" event={"ID":"707073bb-7800-4be5-93d5-87d02b168c1e","Type":"ContainerStarted","Data":"b19a17ac52c4e1eb29a07a332870c254a277c82a955f0f4ff0f86692da33c792"}
Sep 08 10:06:49.090305 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:49.090266    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs" event={"ID":"707073bb-7800-4be5-93d5-87d02b168c1e","Type":"ContainerStarted","Data":"c6e1285d9d96538dd066fff472a58956484533c4fe59743adf065498fd7a53c2"}
Sep 08 10:06:50.096499 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:50.096468    2559 generic.go:356] "Generic (PLEG): container finished" podID="707073bb-7800-4be5-93d5-87d02b168c1e" containerID="c6e1285d9d96538dd066fff472a58956484533c4fe59743adf065498fd7a53c2" exitCode=0
Sep 08 10:06:50.096858 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:50.096506    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs" event={"ID":"707073bb-7800-4be5-93d5-87d02b168c1e","Type":"ContainerDied","Data":"c6e1285d9d96538dd066fff472a58956484533c4fe59743adf065498fd7a53c2"}
Sep 08 10:06:52.106205 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:52.106117    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs" event={"ID":"707073bb-7800-4be5-93d5-87d02b168c1e","Type":"ContainerStarted","Data":"c96b57ee03b449da6bdfc9b53ddfcf57f3075b1b71bc4e79d65e00f50d3ff6cb"}
Sep 08 10:06:52.106611 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:52.106213    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:06:52.129110 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:06:52.129065    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs" podStartSLOduration=2.196716483 podStartE2EDuration="9.129049675s" totalImagesPullingTime="5.846557018s" totalInitContainerRuntime="1.085776174s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:06:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:06:52.12694686 +0000 UTC m=+331.968171290" watchObservedRunningTime="2026-09-08 10:06:52.129049675 +0000 UTC m=+331.970274058"
Sep 08 10:07:02.241382 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.241345    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"]
Sep 08 10:07:02.251259 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.251207    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:02.253328 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.253300    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"]
Sep 08 10:07:02.254458 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.254439    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 10:07:02.254661 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.254639    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 10:07:02.254776 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.254749    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 10:07:02.254891 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.254657    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tssx4\""
Sep 08 10:07:02.254891 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.254681    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 10:07:02.254891 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.254701    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 10:07:02.378651 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.378624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3196d75c-584d-48d5-bf15-8604c9616f7a-cert\") pod \"lws-controller-manager-86bd4f8cd9-5gbr5\" (UID: \"3196d75c-584d-48d5-bf15-8604c9616f7a\") " pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:02.378772 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.378654    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3196d75c-584d-48d5-bf15-8604c9616f7a-manager-config\") pod \"lws-controller-manager-86bd4f8cd9-5gbr5\" (UID: \"3196d75c-584d-48d5-bf15-8604c9616f7a\") " pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:02.378772 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.378671    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3196d75c-584d-48d5-bf15-8604c9616f7a-metrics-cert\") pod \"lws-controller-manager-86bd4f8cd9-5gbr5\" (UID: \"3196d75c-584d-48d5-bf15-8604c9616f7a\") " pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:02.378772 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.378754    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th9sw\" (UniqueName: \"kubernetes.io/projected/3196d75c-584d-48d5-bf15-8604c9616f7a-kube-api-access-th9sw\") pod \"lws-controller-manager-86bd4f8cd9-5gbr5\" (UID: \"3196d75c-584d-48d5-bf15-8604c9616f7a\") " pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:02.479822 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.479789    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th9sw\" (UniqueName: \"kubernetes.io/projected/3196d75c-584d-48d5-bf15-8604c9616f7a-kube-api-access-th9sw\") pod \"lws-controller-manager-86bd4f8cd9-5gbr5\" (UID: \"3196d75c-584d-48d5-bf15-8604c9616f7a\") " pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:02.479961 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.479856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3196d75c-584d-48d5-bf15-8604c9616f7a-cert\") pod \"lws-controller-manager-86bd4f8cd9-5gbr5\" (UID: \"3196d75c-584d-48d5-bf15-8604c9616f7a\") " pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:02.479961 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.479884    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3196d75c-584d-48d5-bf15-8604c9616f7a-manager-config\") pod \"lws-controller-manager-86bd4f8cd9-5gbr5\" (UID: \"3196d75c-584d-48d5-bf15-8604c9616f7a\") " pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:02.479961 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.479906    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3196d75c-584d-48d5-bf15-8604c9616f7a-metrics-cert\") pod \"lws-controller-manager-86bd4f8cd9-5gbr5\" (UID: \"3196d75c-584d-48d5-bf15-8604c9616f7a\") " pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:02.480505 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.480483    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3196d75c-584d-48d5-bf15-8604c9616f7a-manager-config\") pod \"lws-controller-manager-86bd4f8cd9-5gbr5\" (UID: \"3196d75c-584d-48d5-bf15-8604c9616f7a\") " pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:02.482521 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.482497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3196d75c-584d-48d5-bf15-8604c9616f7a-metrics-cert\") pod \"lws-controller-manager-86bd4f8cd9-5gbr5\" (UID: \"3196d75c-584d-48d5-bf15-8604c9616f7a\") " pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:02.482613 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.482497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3196d75c-584d-48d5-bf15-8604c9616f7a-cert\") pod \"lws-controller-manager-86bd4f8cd9-5gbr5\" (UID: \"3196d75c-584d-48d5-bf15-8604c9616f7a\") " pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:02.487750 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.487731    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th9sw\" (UniqueName: \"kubernetes.io/projected/3196d75c-584d-48d5-bf15-8604c9616f7a-kube-api-access-th9sw\") pod \"lws-controller-manager-86bd4f8cd9-5gbr5\" (UID: \"3196d75c-584d-48d5-bf15-8604c9616f7a\") " pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:02.561080 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.561027    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:02.831357 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:02.831335    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"]
Sep 08 10:07:02.833550 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:07:02.833524    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3196d75c_584d_48d5_bf15_8604c9616f7a.slice/crio-e73cf2775b3582277468d3c716248349941bc23bf240071d00ee1ab4bdc9994a WatchSource:0}: Error finding container e73cf2775b3582277468d3c716248349941bc23bf240071d00ee1ab4bdc9994a: Status 404 returned error can't find the container with id e73cf2775b3582277468d3c716248349941bc23bf240071d00ee1ab4bdc9994a
Sep 08 10:07:03.113014 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:03.112951    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56b6d9c7b-hvsvs"
Sep 08 10:07:03.156692 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:03.156663    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5" event={"ID":"3196d75c-584d-48d5-bf15-8604c9616f7a","Type":"ContainerStarted","Data":"e73cf2775b3582277468d3c716248349941bc23bf240071d00ee1ab4bdc9994a"}
Sep 08 10:07:05.164902 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.158367    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"]
Sep 08 10:07:05.168653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.168628    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"
Sep 08 10:07:05.174162 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.174125    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bmmm6\""
Sep 08 10:07:05.175728 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.175700    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 10:07:05.187946 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.187925    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 10:07:05.200320 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.200294    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"]
Sep 08 10:07:05.305036 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.305005    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tpdt\" (UniqueName: \"kubernetes.io/projected/09c3d1b2-6a41-44d0-8413-9f862a16e6ab-kube-api-access-4tpdt\") pod \"kube-auth-proxy-77c9fc4cdc-fvm6d\" (UID: \"09c3d1b2-6a41-44d0-8413-9f862a16e6ab\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"
Sep 08 10:07:05.305212 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.305069    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09c3d1b2-6a41-44d0-8413-9f862a16e6ab-tls-certs\") pod \"kube-auth-proxy-77c9fc4cdc-fvm6d\" (UID: \"09c3d1b2-6a41-44d0-8413-9f862a16e6ab\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"
Sep 08 10:07:05.305212 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.305182    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09c3d1b2-6a41-44d0-8413-9f862a16e6ab-tmp\") pod \"kube-auth-proxy-77c9fc4cdc-fvm6d\" (UID: \"09c3d1b2-6a41-44d0-8413-9f862a16e6ab\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"
Sep 08 10:07:05.405977 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.405945    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tpdt\" (UniqueName: \"kubernetes.io/projected/09c3d1b2-6a41-44d0-8413-9f862a16e6ab-kube-api-access-4tpdt\") pod \"kube-auth-proxy-77c9fc4cdc-fvm6d\" (UID: \"09c3d1b2-6a41-44d0-8413-9f862a16e6ab\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"
Sep 08 10:07:05.406193 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.406139    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09c3d1b2-6a41-44d0-8413-9f862a16e6ab-tls-certs\") pod \"kube-auth-proxy-77c9fc4cdc-fvm6d\" (UID: \"09c3d1b2-6a41-44d0-8413-9f862a16e6ab\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"
Sep 08 10:07:05.406330 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.406282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09c3d1b2-6a41-44d0-8413-9f862a16e6ab-tmp\") pod \"kube-auth-proxy-77c9fc4cdc-fvm6d\" (UID: \"09c3d1b2-6a41-44d0-8413-9f862a16e6ab\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"
Sep 08 10:07:05.406330 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:07:05.406309    2559 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 08 10:07:05.406445 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:07:05.406390    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09c3d1b2-6a41-44d0-8413-9f862a16e6ab-tls-certs podName:09c3d1b2-6a41-44d0-8413-9f862a16e6ab nodeName:}" failed. No retries permitted until 2026-09-08 10:07:05.906369417 +0000 UTC m=+345.747593783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09c3d1b2-6a41-44d0-8413-9f862a16e6ab-tls-certs") pod "kube-auth-proxy-77c9fc4cdc-fvm6d" (UID: "09c3d1b2-6a41-44d0-8413-9f862a16e6ab") : secret "kube-auth-proxy-tls" not found
Sep 08 10:07:05.408624 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.408599    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09c3d1b2-6a41-44d0-8413-9f862a16e6ab-tmp\") pod \"kube-auth-proxy-77c9fc4cdc-fvm6d\" (UID: \"09c3d1b2-6a41-44d0-8413-9f862a16e6ab\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"
Sep 08 10:07:05.423725 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.423675    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tpdt\" (UniqueName: \"kubernetes.io/projected/09c3d1b2-6a41-44d0-8413-9f862a16e6ab-kube-api-access-4tpdt\") pod \"kube-auth-proxy-77c9fc4cdc-fvm6d\" (UID: \"09c3d1b2-6a41-44d0-8413-9f862a16e6ab\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"
Sep 08 10:07:05.908984 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.908957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09c3d1b2-6a41-44d0-8413-9f862a16e6ab-tls-certs\") pod \"kube-auth-proxy-77c9fc4cdc-fvm6d\" (UID: \"09c3d1b2-6a41-44d0-8413-9f862a16e6ab\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"
Sep 08 10:07:05.911216 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:05.911196    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09c3d1b2-6a41-44d0-8413-9f862a16e6ab-tls-certs\") pod \"kube-auth-proxy-77c9fc4cdc-fvm6d\" (UID: \"09c3d1b2-6a41-44d0-8413-9f862a16e6ab\") " pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"
Sep 08 10:07:06.079368 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:06.079340    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"
Sep 08 10:07:06.176403 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:06.176331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5" event={"ID":"3196d75c-584d-48d5-bf15-8604c9616f7a","Type":"ContainerStarted","Data":"bc016bbd6a6a9a385fd1e6745a0e1228f6098980a632a3dce9e5205c7a8b7e31"}
Sep 08 10:07:06.176807 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:06.176472    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:07:06.243875 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:06.243797    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5" podStartSLOduration=1.536186689 podStartE2EDuration="4.243781735s" totalImagesPullingTime="2.707595046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:07:06.242994023 +0000 UTC m=+346.084218427" watchObservedRunningTime="2026-09-08 10:07:06.243781735 +0000 UTC m=+346.085006195"
Sep 08 10:07:06.358213 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:06.358189    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d"]
Sep 08 10:07:06.358986 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:07:06.358958    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c3d1b2_6a41_44d0_8413_9f862a16e6ab.slice/crio-1c4694832851d0ba2e06716ead507986e8cca39b9f8ffa093be357805b5b6d15 WatchSource:0}: Error finding container 1c4694832851d0ba2e06716ead507986e8cca39b9f8ffa093be357805b5b6d15: Status 404 returned error can't find the container with id 1c4694832851d0ba2e06716ead507986e8cca39b9f8ffa093be357805b5b6d15
Sep 08 10:07:07.181475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:07.181436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d" event={"ID":"09c3d1b2-6a41-44d0-8413-9f862a16e6ab","Type":"ContainerStarted","Data":"1c4694832851d0ba2e06716ead507986e8cca39b9f8ffa093be357805b5b6d15"}
Sep 08 10:07:11.202032 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:11.201998    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d" event={"ID":"09c3d1b2-6a41-44d0-8413-9f862a16e6ab","Type":"ContainerStarted","Data":"3497e9183d21ed00d0536fd8f3cbc348c973adb7a7d24935809a63a4ba53b72b"}
Sep 08 10:07:11.218877 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:11.218831    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77c9fc4cdc-fvm6d" podStartSLOduration=1.844867409 podStartE2EDuration="6.218816653s" totalImagesPullingTime="4.373949244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:07:11.217997393 +0000 UTC m=+351.059221798" watchObservedRunningTime="2026-09-08 10:07:11.218816653 +0000 UTC m=+351.060041036"
Sep 08 10:07:17.183989 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:07:17.183908    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-86bd4f8cd9-5gbr5"
Sep 08 10:09:03.058677 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.058639    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"]
Sep 08 10:09:03.061988 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.061972    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"
Sep 08 10:09:03.064712 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.064694    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 10:09:03.064981 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.064968    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 10:09:03.065409 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.065392    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pz647\""
Sep 08 10:09:03.084257 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.084218    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"]
Sep 08 10:09:03.149850 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.149826    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gstkw\" (UniqueName: \"kubernetes.io/projected/c37b4b46-4c9e-48f7-bb14-93e652b741b6-kube-api-access-gstkw\") pod \"kuadrant-operator-controller-manager-96db98c47-68mlm\" (UID: \"c37b4b46-4c9e-48f7-bb14-93e652b741b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"
Sep 08 10:09:03.149959 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.149862    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c37b4b46-4c9e-48f7-bb14-93e652b741b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-68mlm\" (UID: \"c37b4b46-4c9e-48f7-bb14-93e652b741b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"
Sep 08 10:09:03.174678 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.174653    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"]
Sep 08 10:09:03.174861 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:09:03.174843    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-gstkw], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm" podUID="c37b4b46-4c9e-48f7-bb14-93e652b741b6"
Sep 08 10:09:03.250555 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.250527    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gstkw\" (UniqueName: \"kubernetes.io/projected/c37b4b46-4c9e-48f7-bb14-93e652b741b6-kube-api-access-gstkw\") pod \"kuadrant-operator-controller-manager-96db98c47-68mlm\" (UID: \"c37b4b46-4c9e-48f7-bb14-93e652b741b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"
Sep 08 10:09:03.250656 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.250562    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c37b4b46-4c9e-48f7-bb14-93e652b741b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-68mlm\" (UID: \"c37b4b46-4c9e-48f7-bb14-93e652b741b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"
Sep 08 10:09:03.250888 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.250873    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c37b4b46-4c9e-48f7-bb14-93e652b741b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-68mlm\" (UID: \"c37b4b46-4c9e-48f7-bb14-93e652b741b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"
Sep 08 10:09:03.287159 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.287129    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gstkw\" (UniqueName: \"kubernetes.io/projected/c37b4b46-4c9e-48f7-bb14-93e652b741b6-kube-api-access-gstkw\") pod \"kuadrant-operator-controller-manager-96db98c47-68mlm\" (UID: \"c37b4b46-4c9e-48f7-bb14-93e652b741b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"
Sep 08 10:09:03.720536 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.720507    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"
Sep 08 10:09:03.727493 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.727474    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"
Sep 08 10:09:03.854721 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.854694    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c37b4b46-4c9e-48f7-bb14-93e652b741b6-extensions-socket-volume\") pod \"c37b4b46-4c9e-48f7-bb14-93e652b741b6\" (UID: \"c37b4b46-4c9e-48f7-bb14-93e652b741b6\") "
Sep 08 10:09:03.854880 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.854747    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gstkw\" (UniqueName: \"kubernetes.io/projected/c37b4b46-4c9e-48f7-bb14-93e652b741b6-kube-api-access-gstkw\") pod \"c37b4b46-4c9e-48f7-bb14-93e652b741b6\" (UID: \"c37b4b46-4c9e-48f7-bb14-93e652b741b6\") "
Sep 08 10:09:03.854931 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.854910    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37b4b46-4c9e-48f7-bb14-93e652b741b6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c37b4b46-4c9e-48f7-bb14-93e652b741b6" (UID: "c37b4b46-4c9e-48f7-bb14-93e652b741b6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:09:03.856841 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.856809    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37b4b46-4c9e-48f7-bb14-93e652b741b6-kube-api-access-gstkw" (OuterVolumeSpecName: "kube-api-access-gstkw") pod "c37b4b46-4c9e-48f7-bb14-93e652b741b6" (UID: "c37b4b46-4c9e-48f7-bb14-93e652b741b6"). InnerVolumeSpecName "kube-api-access-gstkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:09:03.956101 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.956070    2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c37b4b46-4c9e-48f7-bb14-93e652b741b6-extensions-socket-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:09:03.956101 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:03.956097    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gstkw\" (UniqueName: \"kubernetes.io/projected/c37b4b46-4c9e-48f7-bb14-93e652b741b6-kube-api-access-gstkw\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:09:04.232909 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:04.232872    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"]
Sep 08 10:09:04.243914 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:04.243887    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"]
Sep 08 10:09:04.307952 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:04.307922    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qztkm"]
Sep 08 10:09:04.311370 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:04.311351    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qztkm"
Sep 08 10:09:04.314067 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:04.314042    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hmgz9\""
Sep 08 10:09:04.329643 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:04.329622    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qztkm"]
Sep 08 10:09:04.460821 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:04.460793    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd2rz\" (UniqueName: \"kubernetes.io/projected/57246cb3-882d-4ba1-ba62-cb04e246d315-kube-api-access-nd2rz\") pod \"limitador-operator-controller-manager-6c549b6d85-qztkm\" (UID: \"57246cb3-882d-4ba1-ba62-cb04e246d315\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qztkm"
Sep 08 10:09:04.561941 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:04.561873    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2rz\" (UniqueName: \"kubernetes.io/projected/57246cb3-882d-4ba1-ba62-cb04e246d315-kube-api-access-nd2rz\") pod \"limitador-operator-controller-manager-6c549b6d85-qztkm\" (UID: \"57246cb3-882d-4ba1-ba62-cb04e246d315\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qztkm"
Sep 08 10:09:04.570905 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:04.570870    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2rz\" (UniqueName: \"kubernetes.io/projected/57246cb3-882d-4ba1-ba62-cb04e246d315-kube-api-access-nd2rz\") pod \"limitador-operator-controller-manager-6c549b6d85-qztkm\" (UID: \"57246cb3-882d-4ba1-ba62-cb04e246d315\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qztkm"
Sep 08 10:09:04.621753 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:04.621726    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qztkm"
Sep 08 10:09:04.638472 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:04.638449    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c37b4b46-4c9e-48f7-bb14-93e652b741b6" path="/var/lib/kubelet/pods/c37b4b46-4c9e-48f7-bb14-93e652b741b6/volumes"
Sep 08 10:09:04.724757 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:04.724732    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-68mlm"
Sep 08 10:09:04.893760 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:04.893613    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qztkm"]
Sep 08 10:09:04.896086 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:09:04.896058    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57246cb3_882d_4ba1_ba62_cb04e246d315.slice/crio-636546f5c06379ef59ced1bd08776e2e6a51b0f23dd439a64c24c8342ccde4e9 WatchSource:0}: Error finding container 636546f5c06379ef59ced1bd08776e2e6a51b0f23dd439a64c24c8342ccde4e9: Status 404 returned error can't find the container with id 636546f5c06379ef59ced1bd08776e2e6a51b0f23dd439a64c24c8342ccde4e9
Sep 08 10:09:05.738819 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:05.738778    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qztkm" event={"ID":"57246cb3-882d-4ba1-ba62-cb04e246d315","Type":"ContainerStarted","Data":"636546f5c06379ef59ced1bd08776e2e6a51b0f23dd439a64c24c8342ccde4e9"}
Sep 08 10:09:07.750559 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:07.750531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qztkm" event={"ID":"57246cb3-882d-4ba1-ba62-cb04e246d315","Type":"ContainerStarted","Data":"e2164f8443c06b40330bb72edae812f00b4b040241f4fe62085a668d8094a218"}
Sep 08 10:09:07.750995 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:07.750660    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qztkm"
Sep 08 10:09:07.772704 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:07.772657    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qztkm" podStartSLOduration=1.685721094 podStartE2EDuration="3.772644517s" totalImagesPullingTime="2.086923423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:09:07.770661486 +0000 UTC m=+467.611885879" watchObservedRunningTime="2026-09-08 10:09:07.772644517 +0000 UTC m=+467.613868899"
Sep 08 10:09:18.757221 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:18.757188    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qztkm"
Sep 08 10:09:45.995725 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:45.995691    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-vxx5h"]
Sep 08 10:09:45.999109 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:45.999093    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-vxx5h"
Sep 08 10:09:46.002765 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.002741    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jmqbz\""
Sep 08 10:09:46.016910 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.016889    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-vxx5h"]
Sep 08 10:09:46.102543 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.102519    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqtsc\" (UniqueName: \"kubernetes.io/projected/74faa6eb-3fa8-4b8c-9298-a89d24d401e6-kube-api-access-mqtsc\") pod \"authorino-b54df94b4-vxx5h\" (UID: \"74faa6eb-3fa8-4b8c-9298-a89d24d401e6\") " pod="kuadrant-system/authorino-b54df94b4-vxx5h"
Sep 08 10:09:46.203155 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.203115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqtsc\" (UniqueName: \"kubernetes.io/projected/74faa6eb-3fa8-4b8c-9298-a89d24d401e6-kube-api-access-mqtsc\") pod \"authorino-b54df94b4-vxx5h\" (UID: \"74faa6eb-3fa8-4b8c-9298-a89d24d401e6\") " pod="kuadrant-system/authorino-b54df94b4-vxx5h"
Sep 08 10:09:46.213948 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.213926    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqtsc\" (UniqueName: \"kubernetes.io/projected/74faa6eb-3fa8-4b8c-9298-a89d24d401e6-kube-api-access-mqtsc\") pod \"authorino-b54df94b4-vxx5h\" (UID: \"74faa6eb-3fa8-4b8c-9298-a89d24d401e6\") " pod="kuadrant-system/authorino-b54df94b4-vxx5h"
Sep 08 10:09:46.324346 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.324290    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-vxx5h"
Sep 08 10:09:46.463481 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.463452    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-k2rzr"]
Sep 08 10:09:46.466768 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.466749    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-k2rzr"
Sep 08 10:09:46.473560 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.473532    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-k2rzr"]
Sep 08 10:09:46.505719 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.505692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2n2h\" (UniqueName: \"kubernetes.io/projected/5498c808-50a1-4fa6-8af8-4888035fe9b2-kube-api-access-w2n2h\") pod \"authorino-c548b5d7d-k2rzr\" (UID: \"5498c808-50a1-4fa6-8af8-4888035fe9b2\") " pod="kuadrant-system/authorino-c548b5d7d-k2rzr"
Sep 08 10:09:46.593836 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.593813    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-vxx5h"]
Sep 08 10:09:46.595432 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:09:46.595409    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74faa6eb_3fa8_4b8c_9298_a89d24d401e6.slice/crio-edd59b6de026b7dffeaee9034f493fc912abdc223fbb981801a0892ab9e911a0 WatchSource:0}: Error finding container edd59b6de026b7dffeaee9034f493fc912abdc223fbb981801a0892ab9e911a0: Status 404 returned error can't find the container with id edd59b6de026b7dffeaee9034f493fc912abdc223fbb981801a0892ab9e911a0
Sep 08 10:09:46.606434 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.606410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2n2h\" (UniqueName: \"kubernetes.io/projected/5498c808-50a1-4fa6-8af8-4888035fe9b2-kube-api-access-w2n2h\") pod \"authorino-c548b5d7d-k2rzr\" (UID: \"5498c808-50a1-4fa6-8af8-4888035fe9b2\") " pod="kuadrant-system/authorino-c548b5d7d-k2rzr"
Sep 08 10:09:46.619225 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.619193    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2n2h\" (UniqueName: \"kubernetes.io/projected/5498c808-50a1-4fa6-8af8-4888035fe9b2-kube-api-access-w2n2h\") pod \"authorino-c548b5d7d-k2rzr\" (UID: \"5498c808-50a1-4fa6-8af8-4888035fe9b2\") " pod="kuadrant-system/authorino-c548b5d7d-k2rzr"
Sep 08 10:09:46.776525 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.776501    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-k2rzr"
Sep 08 10:09:46.925293 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:46.925262    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-vxx5h" event={"ID":"74faa6eb-3fa8-4b8c-9298-a89d24d401e6","Type":"ContainerStarted","Data":"edd59b6de026b7dffeaee9034f493fc912abdc223fbb981801a0892ab9e911a0"}
Sep 08 10:09:47.050898 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:47.050874    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-k2rzr"]
Sep 08 10:09:47.053152 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:09:47.053129    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5498c808_50a1_4fa6_8af8_4888035fe9b2.slice/crio-b105c60c333f00b7aa2164d6d821d1a724144deee1beb0be65bfd28ece6de15b WatchSource:0}: Error finding container b105c60c333f00b7aa2164d6d821d1a724144deee1beb0be65bfd28ece6de15b: Status 404 returned error can't find the container with id b105c60c333f00b7aa2164d6d821d1a724144deee1beb0be65bfd28ece6de15b
Sep 08 10:09:47.931578 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:47.931543    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-k2rzr" event={"ID":"5498c808-50a1-4fa6-8af8-4888035fe9b2","Type":"ContainerStarted","Data":"b105c60c333f00b7aa2164d6d821d1a724144deee1beb0be65bfd28ece6de15b"}
Sep 08 10:09:49.942427 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:49.942391    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-vxx5h" event={"ID":"74faa6eb-3fa8-4b8c-9298-a89d24d401e6","Type":"ContainerStarted","Data":"8dca1878532f0eeb72d7d4c78e19a6edb5e04b477bd12c4c03f18dd72a6fc0bb"}
Sep 08 10:09:49.943997 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:49.943970    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-k2rzr" event={"ID":"5498c808-50a1-4fa6-8af8-4888035fe9b2","Type":"ContainerStarted","Data":"8b9ede988aa9247f66a138d28fd0867eb4127a9cf326bd56d0621811c9e52f61"}
Sep 08 10:09:49.959984 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:49.959939    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-vxx5h" podStartSLOduration=2.341695303 podStartE2EDuration="4.959925498s" totalImagesPullingTime="2.618230195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:09:49.958304663 +0000 UTC m=+509.799529045" watchObservedRunningTime="2026-09-08 10:09:49.959925498 +0000 UTC m=+509.801149882"
Sep 08 10:09:49.975030 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:49.974989    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-k2rzr" podStartSLOduration=1.8140797979999999 podStartE2EDuration="3.974975127s" totalImagesPullingTime="2.160895329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:09:49.973678701 +0000 UTC m=+509.814903096" watchObservedRunningTime="2026-09-08 10:09:49.974975127 +0000 UTC m=+509.816199510"
Sep 08 10:09:50.002768 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:50.002740    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-vxx5h"]
Sep 08 10:09:51.952942 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:51.952899    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-vxx5h" podUID="74faa6eb-3fa8-4b8c-9298-a89d24d401e6" containerName="authorino" containerID="cri-o://8dca1878532f0eeb72d7d4c78e19a6edb5e04b477bd12c4c03f18dd72a6fc0bb" gracePeriod=30
Sep 08 10:09:52.191516 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:52.191493    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-vxx5h"
Sep 08 10:09:52.258940 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:52.258876    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqtsc\" (UniqueName: \"kubernetes.io/projected/74faa6eb-3fa8-4b8c-9298-a89d24d401e6-kube-api-access-mqtsc\") pod \"74faa6eb-3fa8-4b8c-9298-a89d24d401e6\" (UID: \"74faa6eb-3fa8-4b8c-9298-a89d24d401e6\") "
Sep 08 10:09:52.260939 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:52.260920    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74faa6eb-3fa8-4b8c-9298-a89d24d401e6-kube-api-access-mqtsc" (OuterVolumeSpecName: "kube-api-access-mqtsc") pod "74faa6eb-3fa8-4b8c-9298-a89d24d401e6" (UID: "74faa6eb-3fa8-4b8c-9298-a89d24d401e6"). InnerVolumeSpecName "kube-api-access-mqtsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:09:52.359609 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:52.359583    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqtsc\" (UniqueName: \"kubernetes.io/projected/74faa6eb-3fa8-4b8c-9298-a89d24d401e6-kube-api-access-mqtsc\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:09:52.958522 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:52.958492    2559 generic.go:356] "Generic (PLEG): container finished" podID="74faa6eb-3fa8-4b8c-9298-a89d24d401e6" containerID="8dca1878532f0eeb72d7d4c78e19a6edb5e04b477bd12c4c03f18dd72a6fc0bb" exitCode=0
Sep 08 10:09:52.958926 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:52.958529    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-vxx5h"
Sep 08 10:09:52.958926 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:52.958566    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-vxx5h" event={"ID":"74faa6eb-3fa8-4b8c-9298-a89d24d401e6","Type":"ContainerDied","Data":"8dca1878532f0eeb72d7d4c78e19a6edb5e04b477bd12c4c03f18dd72a6fc0bb"}
Sep 08 10:09:52.958926 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:52.958595    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-vxx5h" event={"ID":"74faa6eb-3fa8-4b8c-9298-a89d24d401e6","Type":"ContainerDied","Data":"edd59b6de026b7dffeaee9034f493fc912abdc223fbb981801a0892ab9e911a0"}
Sep 08 10:09:52.958926 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:52.958612    2559 scope.go:117] "RemoveContainer" containerID="8dca1878532f0eeb72d7d4c78e19a6edb5e04b477bd12c4c03f18dd72a6fc0bb"
Sep 08 10:09:52.966584 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:52.966568    2559 scope.go:117] "RemoveContainer" containerID="8dca1878532f0eeb72d7d4c78e19a6edb5e04b477bd12c4c03f18dd72a6fc0bb"
Sep 08 10:09:52.966855 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:09:52.966833    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dca1878532f0eeb72d7d4c78e19a6edb5e04b477bd12c4c03f18dd72a6fc0bb\": container with ID starting with 8dca1878532f0eeb72d7d4c78e19a6edb5e04b477bd12c4c03f18dd72a6fc0bb not found: ID does not exist" containerID="8dca1878532f0eeb72d7d4c78e19a6edb5e04b477bd12c4c03f18dd72a6fc0bb"
Sep 08 10:09:52.966898 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:52.966863    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dca1878532f0eeb72d7d4c78e19a6edb5e04b477bd12c4c03f18dd72a6fc0bb"} err="failed to get container status \"8dca1878532f0eeb72d7d4c78e19a6edb5e04b477bd12c4c03f18dd72a6fc0bb\": rpc error: code = NotFound desc = could not find container \"8dca1878532f0eeb72d7d4c78e19a6edb5e04b477bd12c4c03f18dd72a6fc0bb\": container with ID starting with 8dca1878532f0eeb72d7d4c78e19a6edb5e04b477bd12c4c03f18dd72a6fc0bb not found: ID does not exist"
Sep 08 10:09:52.974720 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:52.974699    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-vxx5h"]
Sep 08 10:09:52.978527 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:52.978508    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-vxx5h"]
Sep 08 10:09:54.636113 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:09:54.636077    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74faa6eb-3fa8-4b8c-9298-a89d24d401e6" path="/var/lib/kubelet/pods/74faa6eb-3fa8-4b8c-9298-a89d24d401e6/volumes"
Sep 08 10:10:14.569555 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.569477    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-826fd"]
Sep 08 10:10:14.569915 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.569831    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74faa6eb-3fa8-4b8c-9298-a89d24d401e6" containerName="authorino"
Sep 08 10:10:14.569915 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.569842    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="74faa6eb-3fa8-4b8c-9298-a89d24d401e6" containerName="authorino"
Sep 08 10:10:14.569915 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.569906    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="74faa6eb-3fa8-4b8c-9298-a89d24d401e6" containerName="authorino"
Sep 08 10:10:14.575633 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.575616    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-826fd"
Sep 08 10:10:14.582598 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.582337    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-826fd"]
Sep 08 10:10:14.656324 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.656291    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh2s4\" (UniqueName: \"kubernetes.io/projected/a81a4a37-00ae-44c8-a1e0-ab6ca0f90030-kube-api-access-qh2s4\") pod \"authorino-5dc7ccb5f-826fd\" (UID: \"a81a4a37-00ae-44c8-a1e0-ab6ca0f90030\") " pod="kuadrant-system/authorino-5dc7ccb5f-826fd"
Sep 08 10:10:14.722684 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.722655    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-826fd"]
Sep 08 10:10:14.722940 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:10:14.722917    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qh2s4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-826fd" podUID="a81a4a37-00ae-44c8-a1e0-ab6ca0f90030"
Sep 08 10:10:14.756786 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.756756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh2s4\" (UniqueName: \"kubernetes.io/projected/a81a4a37-00ae-44c8-a1e0-ab6ca0f90030-kube-api-access-qh2s4\") pod \"authorino-5dc7ccb5f-826fd\" (UID: \"a81a4a37-00ae-44c8-a1e0-ab6ca0f90030\") " pod="kuadrant-system/authorino-5dc7ccb5f-826fd"
Sep 08 10:10:14.758340 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.758320    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-nncsm"]
Sep 08 10:10:14.761487 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.761473    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-nncsm"
Sep 08 10:10:14.767110 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.767090    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh2s4\" (UniqueName: \"kubernetes.io/projected/a81a4a37-00ae-44c8-a1e0-ab6ca0f90030-kube-api-access-qh2s4\") pod \"authorino-5dc7ccb5f-826fd\" (UID: \"a81a4a37-00ae-44c8-a1e0-ab6ca0f90030\") " pod="kuadrant-system/authorino-5dc7ccb5f-826fd"
Sep 08 10:10:14.767500 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.767485    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 08 10:10:14.775146 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.775127    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-nncsm"]
Sep 08 10:10:14.842189 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.842129    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-nncsm"]
Sep 08 10:10:14.842393 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:10:14.842376    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-p2x6w tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-p2x6w tls-cert]: context canceled" pod="kuadrant-system/authorino-687bd45585-nncsm" podUID="52a3590f-8acf-4db5-bd86-30276e419396"
Sep 08 10:10:14.857989 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.857965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52a3590f-8acf-4db5-bd86-30276e419396-tls-cert\") pod \"authorino-687bd45585-nncsm\" (UID: \"52a3590f-8acf-4db5-bd86-30276e419396\") " pod="kuadrant-system/authorino-687bd45585-nncsm"
Sep 08 10:10:14.858071 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.857999    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2x6w\" (UniqueName: \"kubernetes.io/projected/52a3590f-8acf-4db5-bd86-30276e419396-kube-api-access-p2x6w\") pod \"authorino-687bd45585-nncsm\" (UID: \"52a3590f-8acf-4db5-bd86-30276e419396\") " pod="kuadrant-system/authorino-687bd45585-nncsm"
Sep 08 10:10:14.886108 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.886083    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-ccf766dd-njldr"]
Sep 08 10:10:14.889485 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.889469    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ccf766dd-njldr"
Sep 08 10:10:14.899688 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.899668    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-ccf766dd-njldr"]
Sep 08 10:10:14.958611 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.958584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns4ls\" (UniqueName: \"kubernetes.io/projected/f0a33369-7576-48df-bb1f-a6531a6a21bc-kube-api-access-ns4ls\") pod \"authorino-ccf766dd-njldr\" (UID: \"f0a33369-7576-48df-bb1f-a6531a6a21bc\") " pod="kuadrant-system/authorino-ccf766dd-njldr"
Sep 08 10:10:14.958714 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.958660    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0a33369-7576-48df-bb1f-a6531a6a21bc-tls-cert\") pod \"authorino-ccf766dd-njldr\" (UID: \"f0a33369-7576-48df-bb1f-a6531a6a21bc\") " pod="kuadrant-system/authorino-ccf766dd-njldr"
Sep 08 10:10:14.958714 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.958698    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52a3590f-8acf-4db5-bd86-30276e419396-tls-cert\") pod \"authorino-687bd45585-nncsm\" (UID: \"52a3590f-8acf-4db5-bd86-30276e419396\") " pod="kuadrant-system/authorino-687bd45585-nncsm"
Sep 08 10:10:14.958785 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.958747    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2x6w\" (UniqueName: \"kubernetes.io/projected/52a3590f-8acf-4db5-bd86-30276e419396-kube-api-access-p2x6w\") pod \"authorino-687bd45585-nncsm\" (UID: \"52a3590f-8acf-4db5-bd86-30276e419396\") " pod="kuadrant-system/authorino-687bd45585-nncsm"
Sep 08 10:10:14.961000 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.960977    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52a3590f-8acf-4db5-bd86-30276e419396-tls-cert\") pod \"authorino-687bd45585-nncsm\" (UID: \"52a3590f-8acf-4db5-bd86-30276e419396\") " pod="kuadrant-system/authorino-687bd45585-nncsm"
Sep 08 10:10:14.968957 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:14.968934    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2x6w\" (UniqueName: \"kubernetes.io/projected/52a3590f-8acf-4db5-bd86-30276e419396-kube-api-access-p2x6w\") pod \"authorino-687bd45585-nncsm\" (UID: \"52a3590f-8acf-4db5-bd86-30276e419396\") " pod="kuadrant-system/authorino-687bd45585-nncsm"
Sep 08 10:10:15.059531 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.059505    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0a33369-7576-48df-bb1f-a6531a6a21bc-tls-cert\") pod \"authorino-ccf766dd-njldr\" (UID: \"f0a33369-7576-48df-bb1f-a6531a6a21bc\") " pod="kuadrant-system/authorino-ccf766dd-njldr"
Sep 08 10:10:15.059641 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.059552    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns4ls\" (UniqueName: \"kubernetes.io/projected/f0a33369-7576-48df-bb1f-a6531a6a21bc-kube-api-access-ns4ls\") pod \"authorino-ccf766dd-njldr\" (UID: \"f0a33369-7576-48df-bb1f-a6531a6a21bc\") " pod="kuadrant-system/authorino-ccf766dd-njldr"
Sep 08 10:10:15.061763 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.061736    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-nncsm"
Sep 08 10:10:15.061846 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.061735    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-826fd"
Sep 08 10:10:15.061939 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.061922    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0a33369-7576-48df-bb1f-a6531a6a21bc-tls-cert\") pod \"authorino-ccf766dd-njldr\" (UID: \"f0a33369-7576-48df-bb1f-a6531a6a21bc\") " pod="kuadrant-system/authorino-ccf766dd-njldr"
Sep 08 10:10:15.070328 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.070303    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns4ls\" (UniqueName: \"kubernetes.io/projected/f0a33369-7576-48df-bb1f-a6531a6a21bc-kube-api-access-ns4ls\") pod \"authorino-ccf766dd-njldr\" (UID: \"f0a33369-7576-48df-bb1f-a6531a6a21bc\") " pod="kuadrant-system/authorino-ccf766dd-njldr"
Sep 08 10:10:15.082213 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.082197    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-nncsm"
Sep 08 10:10:15.085150 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.085135    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-826fd"
Sep 08 10:10:15.160232 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.160210    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh2s4\" (UniqueName: \"kubernetes.io/projected/a81a4a37-00ae-44c8-a1e0-ab6ca0f90030-kube-api-access-qh2s4\") pod \"a81a4a37-00ae-44c8-a1e0-ab6ca0f90030\" (UID: \"a81a4a37-00ae-44c8-a1e0-ab6ca0f90030\") "
Sep 08 10:10:15.160353 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.160284    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2x6w\" (UniqueName: \"kubernetes.io/projected/52a3590f-8acf-4db5-bd86-30276e419396-kube-api-access-p2x6w\") pod \"52a3590f-8acf-4db5-bd86-30276e419396\" (UID: \"52a3590f-8acf-4db5-bd86-30276e419396\") "
Sep 08 10:10:15.160353 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.160322    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52a3590f-8acf-4db5-bd86-30276e419396-tls-cert\") pod \"52a3590f-8acf-4db5-bd86-30276e419396\" (UID: \"52a3590f-8acf-4db5-bd86-30276e419396\") "
Sep 08 10:10:15.162019 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.161997    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81a4a37-00ae-44c8-a1e0-ab6ca0f90030-kube-api-access-qh2s4" (OuterVolumeSpecName: "kube-api-access-qh2s4") pod "a81a4a37-00ae-44c8-a1e0-ab6ca0f90030" (UID: "a81a4a37-00ae-44c8-a1e0-ab6ca0f90030"). InnerVolumeSpecName "kube-api-access-qh2s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:10:15.162108 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.162069    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a3590f-8acf-4db5-bd86-30276e419396-kube-api-access-p2x6w" (OuterVolumeSpecName: "kube-api-access-p2x6w") pod "52a3590f-8acf-4db5-bd86-30276e419396" (UID: "52a3590f-8acf-4db5-bd86-30276e419396"). InnerVolumeSpecName "kube-api-access-p2x6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:10:15.162170 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.162138    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a3590f-8acf-4db5-bd86-30276e419396-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "52a3590f-8acf-4db5-bd86-30276e419396" (UID: "52a3590f-8acf-4db5-bd86-30276e419396"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:10:15.198214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.198191    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ccf766dd-njldr"
Sep 08 10:10:15.261029 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.261001    2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52a3590f-8acf-4db5-bd86-30276e419396-tls-cert\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:10:15.261139 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.261034    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh2s4\" (UniqueName: \"kubernetes.io/projected/a81a4a37-00ae-44c8-a1e0-ab6ca0f90030-kube-api-access-qh2s4\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:10:15.261139 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.261050    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2x6w\" (UniqueName: \"kubernetes.io/projected/52a3590f-8acf-4db5-bd86-30276e419396-kube-api-access-p2x6w\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:10:15.469568 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:15.469478    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-ccf766dd-njldr"]
Sep 08 10:10:15.472945 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:10:15.472916    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0a33369_7576_48df_bb1f_a6531a6a21bc.slice/crio-7dcd94beda7a9d198df937fcc95da9e848bfe6afd76c95ed449cf9e63809e763 WatchSource:0}: Error finding container 7dcd94beda7a9d198df937fcc95da9e848bfe6afd76c95ed449cf9e63809e763: Status 404 returned error can't find the container with id 7dcd94beda7a9d198df937fcc95da9e848bfe6afd76c95ed449cf9e63809e763
Sep 08 10:10:16.068177 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.068144    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ccf766dd-njldr" event={"ID":"f0a33369-7576-48df-bb1f-a6531a6a21bc","Type":"ContainerStarted","Data":"9d4debefa97b89e5fb9c32bc9a9b5db5c59b18783c3742d56f4885d28b1aadaf"}
Sep 08 10:10:16.068557 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.068191    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ccf766dd-njldr" event={"ID":"f0a33369-7576-48df-bb1f-a6531a6a21bc","Type":"ContainerStarted","Data":"7dcd94beda7a9d198df937fcc95da9e848bfe6afd76c95ed449cf9e63809e763"}
Sep 08 10:10:16.068557 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.068264    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-826fd"
Sep 08 10:10:16.068557 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.068412    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-nncsm"
Sep 08 10:10:16.092466 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.092418    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-ccf766dd-njldr" podStartSLOduration=1.5686808060000001 podStartE2EDuration="2.092396819s" totalImagesPullingTime="523.716013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:10:16.089413052 +0000 UTC m=+535.930637435" watchObservedRunningTime="2026-09-08 10:10:16.092396819 +0000 UTC m=+535.933621202"
Sep 08 10:10:16.138392 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.138364    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-826fd"]
Sep 08 10:10:16.147513 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.147487    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-k2rzr"]
Sep 08 10:10:16.147702 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.147677    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-k2rzr" podUID="5498c808-50a1-4fa6-8af8-4888035fe9b2" containerName="authorino" containerID="cri-o://8b9ede988aa9247f66a138d28fd0867eb4127a9cf326bd56d0621811c9e52f61" gracePeriod=30
Sep 08 10:10:16.149991 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.149964    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-826fd"]
Sep 08 10:10:16.185370 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.185344    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-nncsm"]
Sep 08 10:10:16.188736 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.188716    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-nncsm"]
Sep 08 10:10:16.377309 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.377285    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-k2rzr"
Sep 08 10:10:16.469323 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.469294    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2n2h\" (UniqueName: \"kubernetes.io/projected/5498c808-50a1-4fa6-8af8-4888035fe9b2-kube-api-access-w2n2h\") pod \"5498c808-50a1-4fa6-8af8-4888035fe9b2\" (UID: \"5498c808-50a1-4fa6-8af8-4888035fe9b2\") "
Sep 08 10:10:16.471318 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.471294    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5498c808-50a1-4fa6-8af8-4888035fe9b2-kube-api-access-w2n2h" (OuterVolumeSpecName: "kube-api-access-w2n2h") pod "5498c808-50a1-4fa6-8af8-4888035fe9b2" (UID: "5498c808-50a1-4fa6-8af8-4888035fe9b2"). InnerVolumeSpecName "kube-api-access-w2n2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:10:16.570265 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.570165    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2n2h\" (UniqueName: \"kubernetes.io/projected/5498c808-50a1-4fa6-8af8-4888035fe9b2-kube-api-access-w2n2h\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:10:16.640308 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.640271    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52a3590f-8acf-4db5-bd86-30276e419396" path="/var/lib/kubelet/pods/52a3590f-8acf-4db5-bd86-30276e419396/volumes"
Sep 08 10:10:16.640573 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:16.640555    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a81a4a37-00ae-44c8-a1e0-ab6ca0f90030" path="/var/lib/kubelet/pods/a81a4a37-00ae-44c8-a1e0-ab6ca0f90030/volumes"
Sep 08 10:10:17.074731 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:17.074690    2559 generic.go:356] "Generic (PLEG): container finished" podID="5498c808-50a1-4fa6-8af8-4888035fe9b2" containerID="8b9ede988aa9247f66a138d28fd0867eb4127a9cf326bd56d0621811c9e52f61" exitCode=0
Sep 08 10:10:17.075129 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:17.074743    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-k2rzr"
Sep 08 10:10:17.075129 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:17.074789    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-k2rzr" event={"ID":"5498c808-50a1-4fa6-8af8-4888035fe9b2","Type":"ContainerDied","Data":"8b9ede988aa9247f66a138d28fd0867eb4127a9cf326bd56d0621811c9e52f61"}
Sep 08 10:10:17.075129 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:17.074832    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-k2rzr" event={"ID":"5498c808-50a1-4fa6-8af8-4888035fe9b2","Type":"ContainerDied","Data":"b105c60c333f00b7aa2164d6d821d1a724144deee1beb0be65bfd28ece6de15b"}
Sep 08 10:10:17.075129 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:17.074853    2559 scope.go:117] "RemoveContainer" containerID="8b9ede988aa9247f66a138d28fd0867eb4127a9cf326bd56d0621811c9e52f61"
Sep 08 10:10:17.083103 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:17.083082    2559 scope.go:117] "RemoveContainer" containerID="8b9ede988aa9247f66a138d28fd0867eb4127a9cf326bd56d0621811c9e52f61"
Sep 08 10:10:17.083394 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:10:17.083375    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9ede988aa9247f66a138d28fd0867eb4127a9cf326bd56d0621811c9e52f61\": container with ID starting with 8b9ede988aa9247f66a138d28fd0867eb4127a9cf326bd56d0621811c9e52f61 not found: ID does not exist" containerID="8b9ede988aa9247f66a138d28fd0867eb4127a9cf326bd56d0621811c9e52f61"
Sep 08 10:10:17.083458 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:17.083402    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9ede988aa9247f66a138d28fd0867eb4127a9cf326bd56d0621811c9e52f61"} err="failed to get container status \"8b9ede988aa9247f66a138d28fd0867eb4127a9cf326bd56d0621811c9e52f61\": rpc error: code = NotFound desc = could not find container \"8b9ede988aa9247f66a138d28fd0867eb4127a9cf326bd56d0621811c9e52f61\": container with ID starting with 8b9ede988aa9247f66a138d28fd0867eb4127a9cf326bd56d0621811c9e52f61 not found: ID does not exist"
Sep 08 10:10:17.101693 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:17.101672    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-k2rzr"]
Sep 08 10:10:17.111657 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:17.111633    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-k2rzr"]
Sep 08 10:10:18.637694 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:18.637657    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5498c808-50a1-4fa6-8af8-4888035fe9b2" path="/var/lib/kubelet/pods/5498c808-50a1-4fa6-8af8-4888035fe9b2/volumes"
Sep 08 10:10:59.390866 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.390818    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7ld5x"]
Sep 08 10:10:59.391485 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.391462    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5498c808-50a1-4fa6-8af8-4888035fe9b2" containerName="authorino"
Sep 08 10:10:59.391671 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.391655    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5498c808-50a1-4fa6-8af8-4888035fe9b2" containerName="authorino"
Sep 08 10:10:59.391901 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.391884    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5498c808-50a1-4fa6-8af8-4888035fe9b2" containerName="authorino"
Sep 08 10:10:59.395345 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.395326    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7ld5x"
Sep 08 10:10:59.401835 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.401789    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vbrdv\""
Sep 08 10:10:59.402054 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.402038    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 08 10:10:59.408690 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.408667    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7ld5x"]
Sep 08 10:10:59.520640 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.520600    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j947\" (UniqueName: \"kubernetes.io/projected/31528300-0a49-409d-b0c4-f3083656cc48-kube-api-access-7j947\") pod \"payload-processing-5b54b7fbb7-7ld5x\" (UID: \"31528300-0a49-409d-b0c4-f3083656cc48\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7ld5x"
Sep 08 10:10:59.520834 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.520753    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31528300-0a49-409d-b0c4-f3083656cc48-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7ld5x\" (UID: \"31528300-0a49-409d-b0c4-f3083656cc48\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7ld5x"
Sep 08 10:10:59.622058 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.622012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31528300-0a49-409d-b0c4-f3083656cc48-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7ld5x\" (UID: \"31528300-0a49-409d-b0c4-f3083656cc48\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7ld5x"
Sep 08 10:10:59.622217 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.622077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j947\" (UniqueName: \"kubernetes.io/projected/31528300-0a49-409d-b0c4-f3083656cc48-kube-api-access-7j947\") pod \"payload-processing-5b54b7fbb7-7ld5x\" (UID: \"31528300-0a49-409d-b0c4-f3083656cc48\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7ld5x"
Sep 08 10:10:59.622723 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.622704    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31528300-0a49-409d-b0c4-f3083656cc48-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7ld5x\" (UID: \"31528300-0a49-409d-b0c4-f3083656cc48\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7ld5x"
Sep 08 10:10:59.637667 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.637635    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j947\" (UniqueName: \"kubernetes.io/projected/31528300-0a49-409d-b0c4-f3083656cc48-kube-api-access-7j947\") pod \"payload-processing-5b54b7fbb7-7ld5x\" (UID: \"31528300-0a49-409d-b0c4-f3083656cc48\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7ld5x"
Sep 08 10:10:59.706465 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.706425    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7ld5x"
Sep 08 10:10:59.998711 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:10:59.998647    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7ld5x"]
Sep 08 10:11:00.001077 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:11:00.001045    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31528300_0a49_409d_b0c4_f3083656cc48.slice/crio-4ef88deff8f5e0d1cd99e56283f0f6196007ea6748a0687f3e0a9ab98dfd1525 WatchSource:0}: Error finding container 4ef88deff8f5e0d1cd99e56283f0f6196007ea6748a0687f3e0a9ab98dfd1525: Status 404 returned error can't find the container with id 4ef88deff8f5e0d1cd99e56283f0f6196007ea6748a0687f3e0a9ab98dfd1525
Sep 08 10:11:00.276725 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:00.276644    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7ld5x" event={"ID":"31528300-0a49-409d-b0c4-f3083656cc48","Type":"ContainerStarted","Data":"4ef88deff8f5e0d1cd99e56283f0f6196007ea6748a0687f3e0a9ab98dfd1525"}
Sep 08 10:11:03.294171 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:03.294135    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7ld5x" event={"ID":"31528300-0a49-409d-b0c4-f3083656cc48","Type":"ContainerStarted","Data":"7b832ba7a3ebff1002dc73b91a7bf82179f5eec9d483248e9893e22633aea2f2"}
Sep 08 10:11:03.294744 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:03.294203    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7ld5x"
Sep 08 10:11:03.325713 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:03.325664    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-7ld5x" podStartSLOduration=1.753153357 podStartE2EDuration="4.325648942s" totalImagesPullingTime="2.572495585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:11:03.323770358 +0000 UTC m=+583.164994741" watchObservedRunningTime="2026-09-08 10:11:03.325648942 +0000 UTC m=+583.166873324"
Sep 08 10:11:14.301140 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:14.301101    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7ld5x"
Sep 08 10:11:20.609559 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:20.609529    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:11:20.612139 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:20.612114    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:11:43.284225 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.284150    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"]
Sep 08 10:11:43.292285 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.292267    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.303800 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.303780    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 08 10:11:43.303883 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.303788    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 10:11:43.307085 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.307071    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 10:11:43.315472 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.315456    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fhwp9\""
Sep 08 10:11:43.365153 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.365129    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5387c48a-c2f8-4ce6-97f7-be088db3ad62-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.365284 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.365170    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvr4\" (UniqueName: \"kubernetes.io/projected/5387c48a-c2f8-4ce6-97f7-be088db3ad62-kube-api-access-kkvr4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.365284 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.365230    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5387c48a-c2f8-4ce6-97f7-be088db3ad62-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.365383 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.365313    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5387c48a-c2f8-4ce6-97f7-be088db3ad62-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.365383 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.365344    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5387c48a-c2f8-4ce6-97f7-be088db3ad62-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.365449 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.365388    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5387c48a-c2f8-4ce6-97f7-be088db3ad62-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.391333 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.391314    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"]
Sep 08 10:11:43.466342 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.466320    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5387c48a-c2f8-4ce6-97f7-be088db3ad62-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.466342 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.466349    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5387c48a-c2f8-4ce6-97f7-be088db3ad62-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.466519 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.466379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5387c48a-c2f8-4ce6-97f7-be088db3ad62-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.466519 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.466431    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5387c48a-c2f8-4ce6-97f7-be088db3ad62-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.466519 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.466447    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvr4\" (UniqueName: \"kubernetes.io/projected/5387c48a-c2f8-4ce6-97f7-be088db3ad62-kube-api-access-kkvr4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.466519 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.466479    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5387c48a-c2f8-4ce6-97f7-be088db3ad62-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.466868 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.466842    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5387c48a-c2f8-4ce6-97f7-be088db3ad62-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.466959 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.466902    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5387c48a-c2f8-4ce6-97f7-be088db3ad62-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.466959 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.466929    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5387c48a-c2f8-4ce6-97f7-be088db3ad62-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.468636 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.468617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5387c48a-c2f8-4ce6-97f7-be088db3ad62-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.469010 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.468991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5387c48a-c2f8-4ce6-97f7-be088db3ad62-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.478526 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.478502    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvr4\" (UniqueName: \"kubernetes.io/projected/5387c48a-c2f8-4ce6-97f7-be088db3ad62-kube-api-access-kkvr4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d\" (UID: \"5387c48a-c2f8-4ce6-97f7-be088db3ad62\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.602166 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.602042    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:43.886835 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:43.886805    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"]
Sep 08 10:11:43.888323 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:11:43.888298    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5387c48a_c2f8_4ce6_97f7_be088db3ad62.slice/crio-acced43de00cfeafe0ed155b599128693a4d058827d90e334c85f6abf18a5b20 WatchSource:0}: Error finding container acced43de00cfeafe0ed155b599128693a4d058827d90e334c85f6abf18a5b20: Status 404 returned error can't find the container with id acced43de00cfeafe0ed155b599128693a4d058827d90e334c85f6abf18a5b20
Sep 08 10:11:44.484582 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:44.484539    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d" event={"ID":"5387c48a-c2f8-4ce6-97f7-be088db3ad62","Type":"ContainerStarted","Data":"acced43de00cfeafe0ed155b599128693a4d058827d90e334c85f6abf18a5b20"}
Sep 08 10:11:45.491114 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:45.491083    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d" event={"ID":"5387c48a-c2f8-4ce6-97f7-be088db3ad62","Type":"ContainerStarted","Data":"e212d367d75082523c8885cbac555551adb7cf9bf2053dd9c2b0c85147bca367"}
Sep 08 10:11:45.517993 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:45.517951    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d" podStartSLOduration=1.445248278 podStartE2EDuration="2.517937273s" totalImagesPullingTime="1.072688995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:11:45.516085412 +0000 UTC m=+625.357309796" watchObservedRunningTime="2026-09-08 10:11:45.517937273 +0000 UTC m=+625.359161656"
Sep 08 10:11:53.602883 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:53.602841    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:53.603319 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:53.602915    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:53.615893 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:53.615865    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:54.542497 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:54.542465    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d"
Sep 08 10:11:59.494013 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.493978    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"]
Sep 08 10:11:59.497361 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.497345    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.499887 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.499866    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 08 10:11:59.524233 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.524208    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"]
Sep 08 10:11:59.610809 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.610779    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8664\" (UniqueName: \"kubernetes.io/projected/45073922-69bd-4ce6-b2f7-667f8e0ef467-kube-api-access-b8664\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.610809 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.610817    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45073922-69bd-4ce6-b2f7-667f8e0ef467-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.611005 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.610835    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45073922-69bd-4ce6-b2f7-667f8e0ef467-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.611005 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.610851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45073922-69bd-4ce6-b2f7-667f8e0ef467-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.611005 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.610924    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45073922-69bd-4ce6-b2f7-667f8e0ef467-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.611005 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.610950    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45073922-69bd-4ce6-b2f7-667f8e0ef467-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.711898 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.711865    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8664\" (UniqueName: \"kubernetes.io/projected/45073922-69bd-4ce6-b2f7-667f8e0ef467-kube-api-access-b8664\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.712019 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.711914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45073922-69bd-4ce6-b2f7-667f8e0ef467-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.712019 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.711967    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45073922-69bd-4ce6-b2f7-667f8e0ef467-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.712019 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.711993    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45073922-69bd-4ce6-b2f7-667f8e0ef467-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.712171 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.712127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45073922-69bd-4ce6-b2f7-667f8e0ef467-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.712223 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.712172    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45073922-69bd-4ce6-b2f7-667f8e0ef467-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.712388 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.712365    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45073922-69bd-4ce6-b2f7-667f8e0ef467-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.712453 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.712415    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45073922-69bd-4ce6-b2f7-667f8e0ef467-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.712521 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.712505    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45073922-69bd-4ce6-b2f7-667f8e0ef467-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.714256 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.714223    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45073922-69bd-4ce6-b2f7-667f8e0ef467-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.714515 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.714499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45073922-69bd-4ce6-b2f7-667f8e0ef467-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.723791 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.723764    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8664\" (UniqueName: \"kubernetes.io/projected/45073922-69bd-4ce6-b2f7-667f8e0ef467-kube-api-access-b8664\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l24xl\" (UID: \"45073922-69bd-4ce6-b2f7-667f8e0ef467\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:11:59.807756 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:11:59.807696    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:12:00.092329 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:00.092302    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"]
Sep 08 10:12:00.094438 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:12:00.094414    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45073922_69bd_4ce6_b2f7_667f8e0ef467.slice/crio-8ba8f67199c93e2ad798b9335ff3272ab845b6d9558a46959f2485ddcd3a0982 WatchSource:0}: Error finding container 8ba8f67199c93e2ad798b9335ff3272ab845b6d9558a46959f2485ddcd3a0982: Status 404 returned error can't find the container with id 8ba8f67199c93e2ad798b9335ff3272ab845b6d9558a46959f2485ddcd3a0982
Sep 08 10:12:00.095889 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:00.095872    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 10:12:00.562416 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:00.562385    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl" event={"ID":"45073922-69bd-4ce6-b2f7-667f8e0ef467","Type":"ContainerStarted","Data":"8cd60661d8b8a2793b2e45f7d16f8908a7120228259034e3305a8198ffb80ab0"}
Sep 08 10:12:00.562767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:00.562437    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl" event={"ID":"45073922-69bd-4ce6-b2f7-667f8e0ef467","Type":"ContainerStarted","Data":"8ba8f67199c93e2ad798b9335ff3272ab845b6d9558a46959f2485ddcd3a0982"}
Sep 08 10:12:00.601618 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:00.601570    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl" podStartSLOduration=1.429240635 podStartE2EDuration="1.601555339s" totalImagesPullingTime="172.314704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:12:00.601476218 +0000 UTC m=+640.442700631" watchObservedRunningTime="2026-09-08 10:12:00.601555339 +0000 UTC m=+640.442779722"
Sep 08 10:12:07.052370 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.052332    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"]
Sep 08 10:12:07.056515 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.056495    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.059091 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.059068    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 08 10:12:07.067238 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.067219    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"]
Sep 08 10:12:07.180465 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.180432    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91e8a04-b137-4873-91f4-4494e1726297-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.180616 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.180474    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91e8a04-b137-4873-91f4-4494e1726297-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.180616 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.180504    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkcgr\" (UniqueName: \"kubernetes.io/projected/e91e8a04-b137-4873-91f4-4494e1726297-kube-api-access-jkcgr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.180616 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.180525    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91e8a04-b137-4873-91f4-4494e1726297-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.180616 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.180541    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91e8a04-b137-4873-91f4-4494e1726297-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.180616 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.180560    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91e8a04-b137-4873-91f4-4494e1726297-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.281368 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.281339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91e8a04-b137-4873-91f4-4494e1726297-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.281501 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.281379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91e8a04-b137-4873-91f4-4494e1726297-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.281501 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.281409    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkcgr\" (UniqueName: \"kubernetes.io/projected/e91e8a04-b137-4873-91f4-4494e1726297-kube-api-access-jkcgr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.281501 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.281434    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91e8a04-b137-4873-91f4-4494e1726297-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.281501 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.281458    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91e8a04-b137-4873-91f4-4494e1726297-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.281501 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.281492    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91e8a04-b137-4873-91f4-4494e1726297-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.281827 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.281801    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91e8a04-b137-4873-91f4-4494e1726297-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.281899 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.281817    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91e8a04-b137-4873-91f4-4494e1726297-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.281899 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.281863    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91e8a04-b137-4873-91f4-4494e1726297-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.283844 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.283817    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91e8a04-b137-4873-91f4-4494e1726297-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.284006 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.283985    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91e8a04-b137-4873-91f4-4494e1726297-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.291174 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.291151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkcgr\" (UniqueName: \"kubernetes.io/projected/e91e8a04-b137-4873-91f4-4494e1726297-kube-api-access-jkcgr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x77c9\" (UID: \"e91e8a04-b137-4873-91f4-4494e1726297\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.365753 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.365690    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:07.637035 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:07.637012    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"]
Sep 08 10:12:07.638724 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:12:07.638698    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode91e8a04_b137_4873_91f4_4494e1726297.slice/crio-79e52aeb2039188a7847d7397b53927da09005c0af5accf0048d5eb5fbe6d68f WatchSource:0}: Error finding container 79e52aeb2039188a7847d7397b53927da09005c0af5accf0048d5eb5fbe6d68f: Status 404 returned error can't find the container with id 79e52aeb2039188a7847d7397b53927da09005c0af5accf0048d5eb5fbe6d68f
Sep 08 10:12:08.604015 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:08.603985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9" event={"ID":"e91e8a04-b137-4873-91f4-4494e1726297","Type":"ContainerStarted","Data":"8778f5def9dcd2879beae76161f3e877a2afa403bdae519a5cd9a9fd21d6c0b5"}
Sep 08 10:12:08.604377 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:08.604023    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9" event={"ID":"e91e8a04-b137-4873-91f4-4494e1726297","Type":"ContainerStarted","Data":"79e52aeb2039188a7847d7397b53927da09005c0af5accf0048d5eb5fbe6d68f"}
Sep 08 10:12:08.626253 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:08.626205    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9" podStartSLOduration=1.436975811 podStartE2EDuration="1.626190444s" totalImagesPullingTime="189.214633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:12:08.625285749 +0000 UTC m=+648.466510136" watchObservedRunningTime="2026-09-08 10:12:08.626190444 +0000 UTC m=+648.467414826"
Sep 08 10:12:09.807790 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:09.807763    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:12:09.808204 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:09.807808    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:12:09.825348 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:09.825314    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:12:10.627942 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:10.627915    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l24xl"
Sep 08 10:12:17.365865 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:17.365832    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:17.366341 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:17.365880    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:17.378400 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:17.378377    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:12:17.657850 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:12:17.657827    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x77c9"
Sep 08 10:13:21.814509 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:21.814432    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-67b97c78bf-bpb97"]
Sep 08 10:13:21.817761 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:21.817741    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67b97c78bf-bpb97"
Sep 08 10:13:21.827498 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:21.827476    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67b97c78bf-bpb97"]
Sep 08 10:13:21.906025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:21.905995    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlpp8\" (UniqueName: \"kubernetes.io/projected/abe11f50-eeac-4076-9669-7e0467148023-kube-api-access-vlpp8\") pod \"authorino-67b97c78bf-bpb97\" (UID: \"abe11f50-eeac-4076-9669-7e0467148023\") " pod="kuadrant-system/authorino-67b97c78bf-bpb97"
Sep 08 10:13:21.906138 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:21.906043    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/abe11f50-eeac-4076-9669-7e0467148023-tls-cert\") pod \"authorino-67b97c78bf-bpb97\" (UID: \"abe11f50-eeac-4076-9669-7e0467148023\") " pod="kuadrant-system/authorino-67b97c78bf-bpb97"
Sep 08 10:13:22.007405 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:22.007366    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlpp8\" (UniqueName: \"kubernetes.io/projected/abe11f50-eeac-4076-9669-7e0467148023-kube-api-access-vlpp8\") pod \"authorino-67b97c78bf-bpb97\" (UID: \"abe11f50-eeac-4076-9669-7e0467148023\") " pod="kuadrant-system/authorino-67b97c78bf-bpb97"
Sep 08 10:13:22.007542 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:22.007428    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/abe11f50-eeac-4076-9669-7e0467148023-tls-cert\") pod \"authorino-67b97c78bf-bpb97\" (UID: \"abe11f50-eeac-4076-9669-7e0467148023\") " pod="kuadrant-system/authorino-67b97c78bf-bpb97"
Sep 08 10:13:22.009767 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:22.009741    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/abe11f50-eeac-4076-9669-7e0467148023-tls-cert\") pod \"authorino-67b97c78bf-bpb97\" (UID: \"abe11f50-eeac-4076-9669-7e0467148023\") " pod="kuadrant-system/authorino-67b97c78bf-bpb97"
Sep 08 10:13:22.015290 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:22.015267    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlpp8\" (UniqueName: \"kubernetes.io/projected/abe11f50-eeac-4076-9669-7e0467148023-kube-api-access-vlpp8\") pod \"authorino-67b97c78bf-bpb97\" (UID: \"abe11f50-eeac-4076-9669-7e0467148023\") " pod="kuadrant-system/authorino-67b97c78bf-bpb97"
Sep 08 10:13:22.126608 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:22.126553    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67b97c78bf-bpb97"
Sep 08 10:13:22.397004 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:22.396981    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67b97c78bf-bpb97"]
Sep 08 10:13:22.399072 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:13:22.399042    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabe11f50_eeac_4076_9669_7e0467148023.slice/crio-5e889f5f9097d350ce35579588a377a95aff0355e08f3f5057616320116fa454 WatchSource:0}: Error finding container 5e889f5f9097d350ce35579588a377a95aff0355e08f3f5057616320116fa454: Status 404 returned error can't find the container with id 5e889f5f9097d350ce35579588a377a95aff0355e08f3f5057616320116fa454
Sep 08 10:13:22.949437 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:22.949400    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67b97c78bf-bpb97" event={"ID":"abe11f50-eeac-4076-9669-7e0467148023","Type":"ContainerStarted","Data":"5e889f5f9097d350ce35579588a377a95aff0355e08f3f5057616320116fa454"}
Sep 08 10:13:26.972664 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:26.972623    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67b97c78bf-bpb97" event={"ID":"abe11f50-eeac-4076-9669-7e0467148023","Type":"ContainerStarted","Data":"685bb97c8819f460d7f85bbd7af7731f9134c300efbc454e6248444552475870"}
Sep 08 10:13:26.995199 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:26.995136    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-67b97c78bf-bpb97" podStartSLOduration=1.964926156 podStartE2EDuration="5.995119538s" totalImagesPullingTime="4.030193382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:13:26.994132943 +0000 UTC m=+726.835357329" watchObservedRunningTime="2026-09-08 10:13:26.995119538 +0000 UTC m=+726.836343921"
Sep 08 10:13:27.025414 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.025371    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-ccf766dd-njldr"]
Sep 08 10:13:27.025623 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.025598    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-ccf766dd-njldr" podUID="f0a33369-7576-48df-bb1f-a6531a6a21bc" containerName="authorino" containerID="cri-o://9d4debefa97b89e5fb9c32bc9a9b5db5c59b18783c3742d56f4885d28b1aadaf" gracePeriod=30
Sep 08 10:13:27.280730 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.280700    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ccf766dd-njldr"
Sep 08 10:13:27.347906 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.347871    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns4ls\" (UniqueName: \"kubernetes.io/projected/f0a33369-7576-48df-bb1f-a6531a6a21bc-kube-api-access-ns4ls\") pod \"f0a33369-7576-48df-bb1f-a6531a6a21bc\" (UID: \"f0a33369-7576-48df-bb1f-a6531a6a21bc\") "
Sep 08 10:13:27.348104 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.348006    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0a33369-7576-48df-bb1f-a6531a6a21bc-tls-cert\") pod \"f0a33369-7576-48df-bb1f-a6531a6a21bc\" (UID: \"f0a33369-7576-48df-bb1f-a6531a6a21bc\") "
Sep 08 10:13:27.350615 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.350576    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a33369-7576-48df-bb1f-a6531a6a21bc-kube-api-access-ns4ls" (OuterVolumeSpecName: "kube-api-access-ns4ls") pod "f0a33369-7576-48df-bb1f-a6531a6a21bc" (UID: "f0a33369-7576-48df-bb1f-a6531a6a21bc"). InnerVolumeSpecName "kube-api-access-ns4ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:13:27.359381 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.359352    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a33369-7576-48df-bb1f-a6531a6a21bc-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f0a33369-7576-48df-bb1f-a6531a6a21bc" (UID: "f0a33369-7576-48df-bb1f-a6531a6a21bc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:13:27.449605 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.449535    2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0a33369-7576-48df-bb1f-a6531a6a21bc-tls-cert\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:13:27.449605 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.449578    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns4ls\" (UniqueName: \"kubernetes.io/projected/f0a33369-7576-48df-bb1f-a6531a6a21bc-kube-api-access-ns4ls\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:13:27.981395 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.981362    2559 generic.go:356] "Generic (PLEG): container finished" podID="f0a33369-7576-48df-bb1f-a6531a6a21bc" containerID="9d4debefa97b89e5fb9c32bc9a9b5db5c59b18783c3742d56f4885d28b1aadaf" exitCode=0
Sep 08 10:13:27.981840 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.981414    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ccf766dd-njldr"
Sep 08 10:13:27.981840 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.981438    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ccf766dd-njldr" event={"ID":"f0a33369-7576-48df-bb1f-a6531a6a21bc","Type":"ContainerDied","Data":"9d4debefa97b89e5fb9c32bc9a9b5db5c59b18783c3742d56f4885d28b1aadaf"}
Sep 08 10:13:27.981840 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.981481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ccf766dd-njldr" event={"ID":"f0a33369-7576-48df-bb1f-a6531a6a21bc","Type":"ContainerDied","Data":"7dcd94beda7a9d198df937fcc95da9e848bfe6afd76c95ed449cf9e63809e763"}
Sep 08 10:13:27.981840 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.981501    2559 scope.go:117] "RemoveContainer" containerID="9d4debefa97b89e5fb9c32bc9a9b5db5c59b18783c3742d56f4885d28b1aadaf"
Sep 08 10:13:27.995171 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.993491    2559 scope.go:117] "RemoveContainer" containerID="9d4debefa97b89e5fb9c32bc9a9b5db5c59b18783c3742d56f4885d28b1aadaf"
Sep 08 10:13:27.995171 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:13:27.993830    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4debefa97b89e5fb9c32bc9a9b5db5c59b18783c3742d56f4885d28b1aadaf\": container with ID starting with 9d4debefa97b89e5fb9c32bc9a9b5db5c59b18783c3742d56f4885d28b1aadaf not found: ID does not exist" containerID="9d4debefa97b89e5fb9c32bc9a9b5db5c59b18783c3742d56f4885d28b1aadaf"
Sep 08 10:13:27.995171 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:27.993862    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4debefa97b89e5fb9c32bc9a9b5db5c59b18783c3742d56f4885d28b1aadaf"} err="failed to get container status \"9d4debefa97b89e5fb9c32bc9a9b5db5c59b18783c3742d56f4885d28b1aadaf\": rpc error: code = NotFound desc = could not find container \"9d4debefa97b89e5fb9c32bc9a9b5db5c59b18783c3742d56f4885d28b1aadaf\": container with ID starting with 9d4debefa97b89e5fb9c32bc9a9b5db5c59b18783c3742d56f4885d28b1aadaf not found: ID does not exist"
Sep 08 10:13:28.029527 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:28.029494    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-ccf766dd-njldr"]
Sep 08 10:13:28.038029 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:28.037999    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-ccf766dd-njldr"]
Sep 08 10:13:28.637163 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:28.637129    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0a33369-7576-48df-bb1f-a6531a6a21bc" path="/var/lib/kubelet/pods/f0a33369-7576-48df-bb1f-a6531a6a21bc/volumes"
Sep 08 10:13:52.346480 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.346446    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"]
Sep 08 10:13:52.346904 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.346831    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a33369-7576-48df-bb1f-a6531a6a21bc" containerName="authorino"
Sep 08 10:13:52.346904 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.346843    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a33369-7576-48df-bb1f-a6531a6a21bc" containerName="authorino"
Sep 08 10:13:52.346904 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.346903    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0a33369-7576-48df-bb1f-a6531a6a21bc" containerName="authorino"
Sep 08 10:13:52.350025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.350004    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"
Sep 08 10:13:52.352502 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.352477    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c28013-dockercfg-gd4nx\""
Sep 08 10:13:52.352592 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.352512    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c28013\""
Sep 08 10:13:52.359808 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.359785    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"]
Sep 08 10:13:52.399077 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.399050    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"]
Sep 08 10:13:52.402469 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.402454    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"
Sep 08 10:13:52.412358 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.412337    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"]
Sep 08 10:13:52.460803 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.460779    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nqnm\" (UniqueName: \"kubernetes.io/projected/0585fb9c-2d5e-48f8-a142-19cf206cbfa6-kube-api-access-7nqnm\") pod \"payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q\" (UID: \"0585fb9c-2d5e-48f8-a142-19cf206cbfa6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"
Sep 08 10:13:52.460915 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.460831    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0585fb9c-2d5e-48f8-a142-19cf206cbfa6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q\" (UID: \"0585fb9c-2d5e-48f8-a142-19cf206cbfa6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"
Sep 08 10:13:52.562232 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.562203    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0585fb9c-2d5e-48f8-a142-19cf206cbfa6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q\" (UID: \"0585fb9c-2d5e-48f8-a142-19cf206cbfa6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"
Sep 08 10:13:52.562370 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.562295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3559dc-7cce-42cd-9fd0-86afd305cd2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9\" (UID: \"3d3559dc-7cce-42cd-9fd0-86afd305cd2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"
Sep 08 10:13:52.562370 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.562316    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nqnm\" (UniqueName: \"kubernetes.io/projected/0585fb9c-2d5e-48f8-a142-19cf206cbfa6-kube-api-access-7nqnm\") pod \"payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q\" (UID: \"0585fb9c-2d5e-48f8-a142-19cf206cbfa6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"
Sep 08 10:13:52.562370 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.562337    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl799\" (UniqueName: \"kubernetes.io/projected/3d3559dc-7cce-42cd-9fd0-86afd305cd2e-kube-api-access-wl799\") pod \"payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9\" (UID: \"3d3559dc-7cce-42cd-9fd0-86afd305cd2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"
Sep 08 10:13:52.562872 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.562853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0585fb9c-2d5e-48f8-a142-19cf206cbfa6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q\" (UID: \"0585fb9c-2d5e-48f8-a142-19cf206cbfa6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"
Sep 08 10:13:52.570835 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.570807    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nqnm\" (UniqueName: \"kubernetes.io/projected/0585fb9c-2d5e-48f8-a142-19cf206cbfa6-kube-api-access-7nqnm\") pod \"payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q\" (UID: \"0585fb9c-2d5e-48f8-a142-19cf206cbfa6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"
Sep 08 10:13:52.660388 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.660367    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"
Sep 08 10:13:52.663930 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.663360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3559dc-7cce-42cd-9fd0-86afd305cd2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9\" (UID: \"3d3559dc-7cce-42cd-9fd0-86afd305cd2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"
Sep 08 10:13:52.663930 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.663406    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl799\" (UniqueName: \"kubernetes.io/projected/3d3559dc-7cce-42cd-9fd0-86afd305cd2e-kube-api-access-wl799\") pod \"payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9\" (UID: \"3d3559dc-7cce-42cd-9fd0-86afd305cd2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"
Sep 08 10:13:52.664287 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.664263    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3559dc-7cce-42cd-9fd0-86afd305cd2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9\" (UID: \"3d3559dc-7cce-42cd-9fd0-86afd305cd2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"
Sep 08 10:13:52.671659 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.671634    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl799\" (UniqueName: \"kubernetes.io/projected/3d3559dc-7cce-42cd-9fd0-86afd305cd2e-kube-api-access-wl799\") pod \"payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9\" (UID: \"3d3559dc-7cce-42cd-9fd0-86afd305cd2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"
Sep 08 10:13:52.712627 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.712597    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"
Sep 08 10:13:52.931586 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.931563    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"]
Sep 08 10:13:52.932723 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:13:52.932701    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0585fb9c_2d5e_48f8_a142_19cf206cbfa6.slice/crio-eaa6aebf397f76ea1aa10932756233b2e1700da1ccd3827bd3af657e6d28b3b7 WatchSource:0}: Error finding container eaa6aebf397f76ea1aa10932756233b2e1700da1ccd3827bd3af657e6d28b3b7: Status 404 returned error can't find the container with id eaa6aebf397f76ea1aa10932756233b2e1700da1ccd3827bd3af657e6d28b3b7
Sep 08 10:13:52.995061 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:52.995032    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"]
Sep 08 10:13:52.996230 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:13:52.996207    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3559dc_7cce_42cd_9fd0_86afd305cd2e.slice/crio-be36a723f1a4e32b3b58469e5bc35af91ddf55c1fb3bf524e77157f29f59b53d WatchSource:0}: Error finding container be36a723f1a4e32b3b58469e5bc35af91ddf55c1fb3bf524e77157f29f59b53d: Status 404 returned error can't find the container with id be36a723f1a4e32b3b58469e5bc35af91ddf55c1fb3bf524e77157f29f59b53d
Sep 08 10:13:53.106998 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:53.106962    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9" event={"ID":"3d3559dc-7cce-42cd-9fd0-86afd305cd2e","Type":"ContainerStarted","Data":"39b49ce63d5774293bfcb85f5506b009fd2bbad90874718eac9bd3aca494586a"}
Sep 08 10:13:53.107166 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:53.107028    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9" event={"ID":"3d3559dc-7cce-42cd-9fd0-86afd305cd2e","Type":"ContainerStarted","Data":"be36a723f1a4e32b3b58469e5bc35af91ddf55c1fb3bf524e77157f29f59b53d"}
Sep 08 10:13:53.107326 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:53.107192    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"
Sep 08 10:13:53.108863 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:53.108838    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q" event={"ID":"0585fb9c-2d5e-48f8-a142-19cf206cbfa6","Type":"ContainerStarted","Data":"a6de3ca1d82ad2da102e4252a22846765b71a69e8adc4d797bcc0e1119ddcb98"}
Sep 08 10:13:53.108964 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:53.108882    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q" event={"ID":"0585fb9c-2d5e-48f8-a142-19cf206cbfa6","Type":"ContainerStarted","Data":"eaa6aebf397f76ea1aa10932756233b2e1700da1ccd3827bd3af657e6d28b3b7"}
Sep 08 10:13:53.108964 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:53.108906    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"
Sep 08 10:13:53.128617 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:53.128567    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9" podStartSLOduration=1.128555324 podStartE2EDuration="1.128555324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:13:53.127550754 +0000 UTC m=+752.968775156" watchObservedRunningTime="2026-09-08 10:13:53.128555324 +0000 UTC m=+752.969779714"
Sep 08 10:13:53.145468 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:13:53.145433    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q" podStartSLOduration=1.145421325 podStartE2EDuration="1.145421325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:13:53.143735023 +0000 UTC m=+752.984959431" watchObservedRunningTime="2026-09-08 10:13:53.145421325 +0000 UTC m=+752.986645707"
Sep 08 10:14:00.844157 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:00.844119    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"]
Sep 08 10:14:00.849107 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:00.849081    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"
Sep 08 10:14:00.851456 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:00.851433    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cbf4760c-dockercfg-xbztf\""
Sep 08 10:14:00.851544 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:00.851441    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cbf4760c\""
Sep 08 10:14:00.863347 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:00.863325    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"]
Sep 08 10:14:00.865093 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:00.865073    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"]
Sep 08 10:14:00.868365 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:00.868349    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"
Sep 08 10:14:00.879474 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:00.879452    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"]
Sep 08 10:14:00.936101 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:00.936077    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2faee22b-8ac7-484e-8d17-df5d0d5b416b-plugins-config-volume\") pod \"payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7\" (UID: \"2faee22b-8ac7-484e-8d17-df5d0d5b416b\") " pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"
Sep 08 10:14:00.936214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:00.936116    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm94b\" (UniqueName: \"kubernetes.io/projected/2faee22b-8ac7-484e-8d17-df5d0d5b416b-kube-api-access-zm94b\") pod \"payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7\" (UID: \"2faee22b-8ac7-484e-8d17-df5d0d5b416b\") " pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"
Sep 08 10:14:01.037134 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.037108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm5d5\" (UniqueName: \"kubernetes.io/projected/458798e1-ac60-463a-88d7-db289022248e-kube-api-access-zm5d5\") pod \"payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l\" (UID: \"458798e1-ac60-463a-88d7-db289022248e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"
Sep 08 10:14:01.037307 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.037203    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2faee22b-8ac7-484e-8d17-df5d0d5b416b-plugins-config-volume\") pod \"payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7\" (UID: \"2faee22b-8ac7-484e-8d17-df5d0d5b416b\") " pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"
Sep 08 10:14:01.037307 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.037232    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm94b\" (UniqueName: \"kubernetes.io/projected/2faee22b-8ac7-484e-8d17-df5d0d5b416b-kube-api-access-zm94b\") pod \"payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7\" (UID: \"2faee22b-8ac7-484e-8d17-df5d0d5b416b\") " pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"
Sep 08 10:14:01.037422 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.037336    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/458798e1-ac60-463a-88d7-db289022248e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l\" (UID: \"458798e1-ac60-463a-88d7-db289022248e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"
Sep 08 10:14:01.037853 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.037833    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2faee22b-8ac7-484e-8d17-df5d0d5b416b-plugins-config-volume\") pod \"payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7\" (UID: \"2faee22b-8ac7-484e-8d17-df5d0d5b416b\") " pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"
Sep 08 10:14:01.046876 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.046855    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm94b\" (UniqueName: \"kubernetes.io/projected/2faee22b-8ac7-484e-8d17-df5d0d5b416b-kube-api-access-zm94b\") pod \"payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7\" (UID: \"2faee22b-8ac7-484e-8d17-df5d0d5b416b\") " pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"
Sep 08 10:14:01.138461 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.138394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/458798e1-ac60-463a-88d7-db289022248e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l\" (UID: \"458798e1-ac60-463a-88d7-db289022248e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"
Sep 08 10:14:01.138461 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.138438    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm5d5\" (UniqueName: \"kubernetes.io/projected/458798e1-ac60-463a-88d7-db289022248e-kube-api-access-zm5d5\") pod \"payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l\" (UID: \"458798e1-ac60-463a-88d7-db289022248e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"
Sep 08 10:14:01.139100 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.139078    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/458798e1-ac60-463a-88d7-db289022248e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l\" (UID: \"458798e1-ac60-463a-88d7-db289022248e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"
Sep 08 10:14:01.153379 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.153350    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm5d5\" (UniqueName: \"kubernetes.io/projected/458798e1-ac60-463a-88d7-db289022248e-kube-api-access-zm5d5\") pod \"payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l\" (UID: \"458798e1-ac60-463a-88d7-db289022248e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"
Sep 08 10:14:01.160082 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.160059    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"
Sep 08 10:14:01.180548 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.180524    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"
Sep 08 10:14:01.440405 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.440360    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"]
Sep 08 10:14:01.443568 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:14:01.443543    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458798e1_ac60_463a_88d7_db289022248e.slice/crio-1e6fd1b44ab5a7ee78d20cf760795deac3fb81f0a0ede8eb922ff4b575905101 WatchSource:0}: Error finding container 1e6fd1b44ab5a7ee78d20cf760795deac3fb81f0a0ede8eb922ff4b575905101: Status 404 returned error can't find the container with id 1e6fd1b44ab5a7ee78d20cf760795deac3fb81f0a0ede8eb922ff4b575905101
Sep 08 10:14:01.467567 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.467547    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"]
Sep 08 10:14:01.482466 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:14:01.482384    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2faee22b_8ac7_484e_8d17_df5d0d5b416b.slice/crio-7ffdecdd77c1b8a76bd5bb79bb3eb4b7f3c62421bab2d17f90f254e0416de31a WatchSource:0}: Error finding container 7ffdecdd77c1b8a76bd5bb79bb3eb4b7f3c62421bab2d17f90f254e0416de31a: Status 404 returned error can't find the container with id 7ffdecdd77c1b8a76bd5bb79bb3eb4b7f3c62421bab2d17f90f254e0416de31a
Sep 08 10:14:01.933698 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.933663    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"]
Sep 08 10:14:01.937315 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.937294    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"
Sep 08 10:14:01.941825 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.941804    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-338aba-dockercfg-zwblg\""
Sep 08 10:14:01.941933 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.941847    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-338aba\""
Sep 08 10:14:01.946032 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.946005    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7fea57-52e5-4edc-bdef-fb5ec70e0da9-plugins-config-volume\") pod \"payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd\" (UID: \"fc7fea57-52e5-4edc-bdef-fb5ec70e0da9\") " pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"
Sep 08 10:14:01.946032 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.946034    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5jbm\" (UniqueName: \"kubernetes.io/projected/fc7fea57-52e5-4edc-bdef-fb5ec70e0da9-kube-api-access-d5jbm\") pod \"payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd\" (UID: \"fc7fea57-52e5-4edc-bdef-fb5ec70e0da9\") " pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"
Sep 08 10:14:01.951793 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.951772    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"]
Sep 08 10:14:01.989737 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.989702    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"]
Sep 08 10:14:01.993800 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:01.993781    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"
Sep 08 10:14:02.005055 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.005034    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"]
Sep 08 10:14:02.047163 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.047138    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlq6d\" (UniqueName: \"kubernetes.io/projected/33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b-kube-api-access-tlq6d\") pod \"payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl\" (UID: \"33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"
Sep 08 10:14:02.047332 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.047193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7fea57-52e5-4edc-bdef-fb5ec70e0da9-plugins-config-volume\") pod \"payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd\" (UID: \"fc7fea57-52e5-4edc-bdef-fb5ec70e0da9\") " pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"
Sep 08 10:14:02.047332 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.047227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5jbm\" (UniqueName: \"kubernetes.io/projected/fc7fea57-52e5-4edc-bdef-fb5ec70e0da9-kube-api-access-d5jbm\") pod \"payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd\" (UID: \"fc7fea57-52e5-4edc-bdef-fb5ec70e0da9\") " pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"
Sep 08 10:14:02.047332 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.047335    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl\" (UID: \"33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"
Sep 08 10:14:02.047975 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.047956    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7fea57-52e5-4edc-bdef-fb5ec70e0da9-plugins-config-volume\") pod \"payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd\" (UID: \"fc7fea57-52e5-4edc-bdef-fb5ec70e0da9\") " pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"
Sep 08 10:14:02.058053 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.058031    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5jbm\" (UniqueName: \"kubernetes.io/projected/fc7fea57-52e5-4edc-bdef-fb5ec70e0da9-kube-api-access-d5jbm\") pod \"payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd\" (UID: \"fc7fea57-52e5-4edc-bdef-fb5ec70e0da9\") " pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"
Sep 08 10:14:02.148328 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.148289    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl\" (UID: \"33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"
Sep 08 10:14:02.148436 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.148415    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlq6d\" (UniqueName: \"kubernetes.io/projected/33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b-kube-api-access-tlq6d\") pod \"payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl\" (UID: \"33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"
Sep 08 10:14:02.148929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.148912    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl\" (UID: \"33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"
Sep 08 10:14:02.157396 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.157372    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlq6d\" (UniqueName: \"kubernetes.io/projected/33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b-kube-api-access-tlq6d\") pod \"payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl\" (UID: \"33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"
Sep 08 10:14:02.158808 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.158783    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7" event={"ID":"2faee22b-8ac7-484e-8d17-df5d0d5b416b","Type":"ContainerStarted","Data":"396505c40bd5ee62a63b4847c217b978842ea552e8ec9312daab9683bed90e26"}
Sep 08 10:14:02.158956 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.158937    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"
Sep 08 10:14:02.159037 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.158979    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7" event={"ID":"2faee22b-8ac7-484e-8d17-df5d0d5b416b","Type":"ContainerStarted","Data":"7ffdecdd77c1b8a76bd5bb79bb3eb4b7f3c62421bab2d17f90f254e0416de31a"}
Sep 08 10:14:02.160594 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.160572    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l" event={"ID":"458798e1-ac60-463a-88d7-db289022248e","Type":"ContainerStarted","Data":"b440511241c14b3befa9b8b22908ebd125dcf0773f3b7d32bee46c815e9258cd"}
Sep 08 10:14:02.160695 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.160618    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"
Sep 08 10:14:02.160695 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.160643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l" event={"ID":"458798e1-ac60-463a-88d7-db289022248e","Type":"ContainerStarted","Data":"1e6fd1b44ab5a7ee78d20cf760795deac3fb81f0a0ede8eb922ff4b575905101"}
Sep 08 10:14:02.178625 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.178587    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7" podStartSLOduration=2.178573755 podStartE2EDuration="2.178573755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:14:02.177130705 +0000 UTC m=+762.018355111" watchObservedRunningTime="2026-09-08 10:14:02.178573755 +0000 UTC m=+762.019798139"
Sep 08 10:14:02.196471 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.196396    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l" podStartSLOduration=2.196383346 podStartE2EDuration="2.196383346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:14:02.194342045 +0000 UTC m=+762.035566440" watchObservedRunningTime="2026-09-08 10:14:02.196383346 +0000 UTC m=+762.037607728"
Sep 08 10:14:02.247930 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.247907    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"
Sep 08 10:14:02.305489 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.304885    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"
Sep 08 10:14:02.526800 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.526778    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"]
Sep 08 10:14:02.528611 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:14:02.528579    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7fea57_52e5_4edc_bdef_fb5ec70e0da9.slice/crio-551cc8bd31c011cc409a294d8c2372147b8f1ce2e0c307369f8847c957843bc3 WatchSource:0}: Error finding container 551cc8bd31c011cc409a294d8c2372147b8f1ce2e0c307369f8847c957843bc3: Status 404 returned error can't find the container with id 551cc8bd31c011cc409a294d8c2372147b8f1ce2e0c307369f8847c957843bc3
Sep 08 10:14:02.591674 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:02.591650    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"]
Sep 08 10:14:02.593548 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:14:02.593525    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f34573_c6bd_4ef8_8c4d_f4b250cb9d5b.slice/crio-c822db6114156c6a71c74920242d549213e3cb12f7e5106cad76dafecb1e8a12 WatchSource:0}: Error finding container c822db6114156c6a71c74920242d549213e3cb12f7e5106cad76dafecb1e8a12: Status 404 returned error can't find the container with id c822db6114156c6a71c74920242d549213e3cb12f7e5106cad76dafecb1e8a12
Sep 08 10:14:03.168675 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:03.168628    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl" event={"ID":"33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b","Type":"ContainerStarted","Data":"273abd33f251dce424e96281de2f52f0508c958d38211b5c061d7729898cf981"}
Sep 08 10:14:03.169123 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:03.168724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl" event={"ID":"33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b","Type":"ContainerStarted","Data":"c822db6114156c6a71c74920242d549213e3cb12f7e5106cad76dafecb1e8a12"}
Sep 08 10:14:03.169123 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:03.169004    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"
Sep 08 10:14:03.170922 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:03.170893    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd" event={"ID":"fc7fea57-52e5-4edc-bdef-fb5ec70e0da9","Type":"ContainerStarted","Data":"eca9b48a877299d2f97201d79e6869583bdc287603de9dd5faef8556fc47f01b"}
Sep 08 10:14:03.171088 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:03.171071    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"
Sep 08 10:14:03.171181 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:03.171123    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd" event={"ID":"fc7fea57-52e5-4edc-bdef-fb5ec70e0da9","Type":"ContainerStarted","Data":"551cc8bd31c011cc409a294d8c2372147b8f1ce2e0c307369f8847c957843bc3"}
Sep 08 10:14:03.188465 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:03.188419    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl" podStartSLOduration=2.188402984 podStartE2EDuration="2.188402984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:14:03.187931793 +0000 UTC m=+763.029156200" watchObservedRunningTime="2026-09-08 10:14:03.188402984 +0000 UTC m=+763.029627370"
Sep 08 10:14:03.209589 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:03.209538    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd" podStartSLOduration=2.209523534 podStartE2EDuration="2.209523534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:14:03.206190866 +0000 UTC m=+763.047415273" watchObservedRunningTime="2026-09-08 10:14:03.209523534 +0000 UTC m=+763.050747922"
Sep 08 10:14:04.115568 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:04.115533    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"
Sep 08 10:14:04.115744 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:04.115636    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"
Sep 08 10:14:13.173007 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:13.172972    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"
Sep 08 10:14:13.173397 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:13.173043    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"
Sep 08 10:14:14.180553 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:14.180524    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"
Sep 08 10:14:14.180951 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:14.180604    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"
Sep 08 10:14:21.196873 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.196832    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"]
Sep 08 10:14:21.201793 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.201771    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"]
Sep 08 10:14:21.201930 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.201887    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"
Sep 08 10:14:21.204861 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.204838    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9e365b\""
Sep 08 10:14:21.205475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.205451    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"
Sep 08 10:14:21.230814 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.230791    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"]
Sep 08 10:14:21.244049 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.244026    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"]
Sep 08 10:14:21.324708 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.324679    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e737b683-5474-48e4-954c-69b3ec23df8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z\" (UID: \"e737b683-5474-48e4-954c-69b3ec23df8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"
Sep 08 10:14:21.324881 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.324716    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtj5z\" (UniqueName: \"kubernetes.io/projected/e737b683-5474-48e4-954c-69b3ec23df8b-kube-api-access-mtj5z\") pod \"payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z\" (UID: \"e737b683-5474-48e4-954c-69b3ec23df8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"
Sep 08 10:14:21.324881 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.324747    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4121c456-ed92-4134-86ac-f6793aa35859-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp\" (UID: \"4121c456-ed92-4134-86ac-f6793aa35859\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"
Sep 08 10:14:21.324881 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.324858    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmwzl\" (UniqueName: \"kubernetes.io/projected/4121c456-ed92-4134-86ac-f6793aa35859-kube-api-access-kmwzl\") pod \"payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp\" (UID: \"4121c456-ed92-4134-86ac-f6793aa35859\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"
Sep 08 10:14:21.425943 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.425907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4121c456-ed92-4134-86ac-f6793aa35859-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp\" (UID: \"4121c456-ed92-4134-86ac-f6793aa35859\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"
Sep 08 10:14:21.426104 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.425969    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmwzl\" (UniqueName: \"kubernetes.io/projected/4121c456-ed92-4134-86ac-f6793aa35859-kube-api-access-kmwzl\") pod \"payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp\" (UID: \"4121c456-ed92-4134-86ac-f6793aa35859\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"
Sep 08 10:14:21.426104 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.426016    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e737b683-5474-48e4-954c-69b3ec23df8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z\" (UID: \"e737b683-5474-48e4-954c-69b3ec23df8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"
Sep 08 10:14:21.426104 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.426033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtj5z\" (UniqueName: \"kubernetes.io/projected/e737b683-5474-48e4-954c-69b3ec23df8b-kube-api-access-mtj5z\") pod \"payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z\" (UID: \"e737b683-5474-48e4-954c-69b3ec23df8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"
Sep 08 10:14:21.426897 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.426874    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4121c456-ed92-4134-86ac-f6793aa35859-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp\" (UID: \"4121c456-ed92-4134-86ac-f6793aa35859\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"
Sep 08 10:14:21.426950 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.426903    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e737b683-5474-48e4-954c-69b3ec23df8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z\" (UID: \"e737b683-5474-48e4-954c-69b3ec23df8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"
Sep 08 10:14:21.443849 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.443816    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmwzl\" (UniqueName: \"kubernetes.io/projected/4121c456-ed92-4134-86ac-f6793aa35859-kube-api-access-kmwzl\") pod \"payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp\" (UID: \"4121c456-ed92-4134-86ac-f6793aa35859\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"
Sep 08 10:14:21.457103 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.457042    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtj5z\" (UniqueName: \"kubernetes.io/projected/e737b683-5474-48e4-954c-69b3ec23df8b-kube-api-access-mtj5z\") pod \"payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z\" (UID: \"e737b683-5474-48e4-954c-69b3ec23df8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"
Sep 08 10:14:21.513239 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.513207    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"
Sep 08 10:14:21.518933 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.518910    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"
Sep 08 10:14:21.793348 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.793320    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"]
Sep 08 10:14:21.795129 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:14:21.795096    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode737b683_5474_48e4_954c_69b3ec23df8b.slice/crio-45aba2c287d523563b5eb356ebe72a2e22e845b104e0bb691f7659934e3fd37e WatchSource:0}: Error finding container 45aba2c287d523563b5eb356ebe72a2e22e845b104e0bb691f7659934e3fd37e: Status 404 returned error can't find the container with id 45aba2c287d523563b5eb356ebe72a2e22e845b104e0bb691f7659934e3fd37e
Sep 08 10:14:21.806476 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:21.806454    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"]
Sep 08 10:14:21.808955 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:14:21.808934    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4121c456_ed92_4134_86ac_f6793aa35859.slice/crio-88d09376dca2cc332dab4bb091cb4be5714bfb3c8295aec74ae79577778dfb3c WatchSource:0}: Error finding container 88d09376dca2cc332dab4bb091cb4be5714bfb3c8295aec74ae79577778dfb3c: Status 404 returned error can't find the container with id 88d09376dca2cc332dab4bb091cb4be5714bfb3c8295aec74ae79577778dfb3c
Sep 08 10:14:22.279843 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:22.279796    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z" event={"ID":"e737b683-5474-48e4-954c-69b3ec23df8b","Type":"ContainerStarted","Data":"83b9537156bf6c31dbb16f507401d00c6a0511e55456faaa1aa257efb0d85cc1"}
Sep 08 10:14:22.280317 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:22.279856    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"
Sep 08 10:14:22.280317 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:22.279891    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z" event={"ID":"e737b683-5474-48e4-954c-69b3ec23df8b","Type":"ContainerStarted","Data":"45aba2c287d523563b5eb356ebe72a2e22e845b104e0bb691f7659934e3fd37e"}
Sep 08 10:14:22.281586 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:22.281556    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp" event={"ID":"4121c456-ed92-4134-86ac-f6793aa35859","Type":"ContainerStarted","Data":"54f103efd37fa8994a368038bf19fc928d18ac7cbdb42c2ee7bc000e94258327"}
Sep 08 10:14:22.281725 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:22.281606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp" event={"ID":"4121c456-ed92-4134-86ac-f6793aa35859","Type":"ContainerStarted","Data":"88d09376dca2cc332dab4bb091cb4be5714bfb3c8295aec74ae79577778dfb3c"}
Sep 08 10:14:22.281725 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:22.281636    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"
Sep 08 10:14:22.301459 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:22.301406    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z" podStartSLOduration=1.301385072 podStartE2EDuration="1.301385072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:14:22.29893545 +0000 UTC m=+782.140159856" watchObservedRunningTime="2026-09-08 10:14:22.301385072 +0000 UTC m=+782.142609456"
Sep 08 10:14:22.318714 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:22.318677    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp" podStartSLOduration=1.318662627 podStartE2EDuration="1.318662627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:14:22.316913594 +0000 UTC m=+782.158137990" watchObservedRunningTime="2026-09-08 10:14:22.318662627 +0000 UTC m=+782.159887010"
Sep 08 10:14:23.812838 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:23.812807    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z"]
Sep 08 10:14:23.817901 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:23.817880    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z"
Sep 08 10:14:23.820112 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:23.820089    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:14:23.820255 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:23.820218    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:14:23.824112 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:23.824090    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z"]
Sep 08 10:14:23.950266 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:23.950216    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e5c2a2-f076-438c-8120-255d5efb750a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z\" (UID: \"66e5c2a2-f076-438c-8120-255d5efb750a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z"
Sep 08 10:14:24.051295 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:24.051233    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e5c2a2-f076-438c-8120-255d5efb750a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z\" (UID: \"66e5c2a2-f076-438c-8120-255d5efb750a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z"
Sep 08 10:14:24.051894 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:24.051876    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e5c2a2-f076-438c-8120-255d5efb750a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z\" (UID: \"66e5c2a2-f076-438c-8120-255d5efb750a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z"
Sep 08 10:14:24.130892 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:24.130823    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z"
Sep 08 10:14:24.410275 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:24.410228    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z"]
Sep 08 10:14:24.412437 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:14:24.412413    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e5c2a2_f076_438c_8120_255d5efb750a.slice/crio-54a74db9783698b8f473c96315e8bcacbc2e40ca9cfaec9087bf4d1d45bbd8fc WatchSource:0}: Error finding container 54a74db9783698b8f473c96315e8bcacbc2e40ca9cfaec9087bf4d1d45bbd8fc: Status 404 returned error can't find the container with id 54a74db9783698b8f473c96315e8bcacbc2e40ca9cfaec9087bf4d1d45bbd8fc
Sep 08 10:14:25.301167 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:25.301136    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z" event={"ID":"66e5c2a2-f076-438c-8120-255d5efb750a","Type":"ContainerStarted","Data":"54a74db9783698b8f473c96315e8bcacbc2e40ca9cfaec9087bf4d1d45bbd8fc"}
Sep 08 10:14:28.319995 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:28.319959    2559 generic.go:356] "Generic (PLEG): container finished" podID="66e5c2a2-f076-438c-8120-255d5efb750a" containerID="3cbe99bcb8723c223d95dce8e22560f8050968b1feb29614e53e0b22ce6c6c5e" exitCode=0
Sep 08 10:14:28.320408 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:28.320059    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z" event={"ID":"66e5c2a2-f076-438c-8120-255d5efb750a","Type":"ContainerDied","Data":"3cbe99bcb8723c223d95dce8e22560f8050968b1feb29614e53e0b22ce6c6c5e"}
Sep 08 10:14:29.474764 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:29.474737    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z"
Sep 08 10:14:29.604983 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:29.604894    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e5c2a2-f076-438c-8120-255d5efb750a-maas-api-service-ca\") pod \"66e5c2a2-f076-438c-8120-255d5efb750a\" (UID: \"66e5c2a2-f076-438c-8120-255d5efb750a\") "
Sep 08 10:14:29.605217 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:29.605195    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e5c2a2-f076-438c-8120-255d5efb750a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "66e5c2a2-f076-438c-8120-255d5efb750a" (UID: "66e5c2a2-f076-438c-8120-255d5efb750a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:14:29.706316 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:29.706287    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e5c2a2-f076-438c-8120-255d5efb750a-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:14:30.332197 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:30.332169    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z"
Sep 08 10:14:30.332388 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:30.332168    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z" event={"ID":"66e5c2a2-f076-438c-8120-255d5efb750a","Type":"ContainerDied","Data":"54a74db9783698b8f473c96315e8bcacbc2e40ca9cfaec9087bf4d1d45bbd8fc"}
Sep 08 10:14:30.332388 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:30.332310    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54a74db9783698b8f473c96315e8bcacbc2e40ca9cfaec9087bf4d1d45bbd8fc"
Sep 08 10:14:33.289305 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:33.289272    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"
Sep 08 10:14:33.289896 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:33.289551    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"
Sep 08 10:14:40.569345 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:40.564766    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"]
Sep 08 10:14:40.569345 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:40.565100    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7" podUID="2faee22b-8ac7-484e-8d17-df5d0d5b416b" containerName="payload-processing" containerID="cri-o://396505c40bd5ee62a63b4847c217b978842ea552e8ec9312daab9683bed90e26" gracePeriod=30
Sep 08 10:14:40.578559 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:40.578530    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"]
Sep 08 10:14:40.578868 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:40.578779    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l" podUID="458798e1-ac60-463a-88d7-db289022248e" containerName="payload-pre-processing" containerID="cri-o://b440511241c14b3befa9b8b22908ebd125dcf0773f3b7d32bee46c815e9258cd" gracePeriod=30
Sep 08 10:14:40.756403 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:40.756371    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z"]
Sep 08 10:14:40.759016 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:40.758993    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbf4760c-zbq6z"]
Sep 08 10:14:40.858118 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:40.858091    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"
Sep 08 10:14:40.861628 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:40.861607    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"
Sep 08 10:14:41.010020 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.009988    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm5d5\" (UniqueName: \"kubernetes.io/projected/458798e1-ac60-463a-88d7-db289022248e-kube-api-access-zm5d5\") pod \"458798e1-ac60-463a-88d7-db289022248e\" (UID: \"458798e1-ac60-463a-88d7-db289022248e\") "
Sep 08 10:14:41.010209 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.010078    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/458798e1-ac60-463a-88d7-db289022248e-plugins-config-volume\") pod \"458798e1-ac60-463a-88d7-db289022248e\" (UID: \"458798e1-ac60-463a-88d7-db289022248e\") "
Sep 08 10:14:41.010209 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.010128    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm94b\" (UniqueName: \"kubernetes.io/projected/2faee22b-8ac7-484e-8d17-df5d0d5b416b-kube-api-access-zm94b\") pod \"2faee22b-8ac7-484e-8d17-df5d0d5b416b\" (UID: \"2faee22b-8ac7-484e-8d17-df5d0d5b416b\") "
Sep 08 10:14:41.010366 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.010214    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2faee22b-8ac7-484e-8d17-df5d0d5b416b-plugins-config-volume\") pod \"2faee22b-8ac7-484e-8d17-df5d0d5b416b\" (UID: \"2faee22b-8ac7-484e-8d17-df5d0d5b416b\") "
Sep 08 10:14:41.010516 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.010485    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/458798e1-ac60-463a-88d7-db289022248e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "458798e1-ac60-463a-88d7-db289022248e" (UID: "458798e1-ac60-463a-88d7-db289022248e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:14:41.010594 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.010557    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2faee22b-8ac7-484e-8d17-df5d0d5b416b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2faee22b-8ac7-484e-8d17-df5d0d5b416b" (UID: "2faee22b-8ac7-484e-8d17-df5d0d5b416b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:14:41.010651 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.010598    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/458798e1-ac60-463a-88d7-db289022248e-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:14:41.012358 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.012328    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458798e1-ac60-463a-88d7-db289022248e-kube-api-access-zm5d5" (OuterVolumeSpecName: "kube-api-access-zm5d5") pod "458798e1-ac60-463a-88d7-db289022248e" (UID: "458798e1-ac60-463a-88d7-db289022248e"). InnerVolumeSpecName "kube-api-access-zm5d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:14:41.012456 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.012388    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2faee22b-8ac7-484e-8d17-df5d0d5b416b-kube-api-access-zm94b" (OuterVolumeSpecName: "kube-api-access-zm94b") pod "2faee22b-8ac7-484e-8d17-df5d0d5b416b" (UID: "2faee22b-8ac7-484e-8d17-df5d0d5b416b"). InnerVolumeSpecName "kube-api-access-zm94b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:14:41.112040 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.111972    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm94b\" (UniqueName: \"kubernetes.io/projected/2faee22b-8ac7-484e-8d17-df5d0d5b416b-kube-api-access-zm94b\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:14:41.112040 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.111997    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2faee22b-8ac7-484e-8d17-df5d0d5b416b-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:14:41.112040 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.112011    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm5d5\" (UniqueName: \"kubernetes.io/projected/458798e1-ac60-463a-88d7-db289022248e-kube-api-access-zm5d5\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:14:41.398196 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.398111    2559 generic.go:356] "Generic (PLEG): container finished" podID="458798e1-ac60-463a-88d7-db289022248e" containerID="b440511241c14b3befa9b8b22908ebd125dcf0773f3b7d32bee46c815e9258cd" exitCode=0
Sep 08 10:14:41.398196 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.398180    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"
Sep 08 10:14:41.398408 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.398262    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l" event={"ID":"458798e1-ac60-463a-88d7-db289022248e","Type":"ContainerDied","Data":"b440511241c14b3befa9b8b22908ebd125dcf0773f3b7d32bee46c815e9258cd"}
Sep 08 10:14:41.398408 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.398312    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l" event={"ID":"458798e1-ac60-463a-88d7-db289022248e","Type":"ContainerDied","Data":"1e6fd1b44ab5a7ee78d20cf760795deac3fb81f0a0ede8eb922ff4b575905101"}
Sep 08 10:14:41.398408 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.398340    2559 scope.go:117] "RemoveContainer" containerID="b440511241c14b3befa9b8b22908ebd125dcf0773f3b7d32bee46c815e9258cd"
Sep 08 10:14:41.399763 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.399734    2559 generic.go:356] "Generic (PLEG): container finished" podID="2faee22b-8ac7-484e-8d17-df5d0d5b416b" containerID="396505c40bd5ee62a63b4847c217b978842ea552e8ec9312daab9683bed90e26" exitCode=0
Sep 08 10:14:41.399873 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.399767    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"
Sep 08 10:14:41.399873 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.399822    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7" event={"ID":"2faee22b-8ac7-484e-8d17-df5d0d5b416b","Type":"ContainerDied","Data":"396505c40bd5ee62a63b4847c217b978842ea552e8ec9312daab9683bed90e26"}
Sep 08 10:14:41.399951 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.399875    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7" event={"ID":"2faee22b-8ac7-484e-8d17-df5d0d5b416b","Type":"ContainerDied","Data":"7ffdecdd77c1b8a76bd5bb79bb3eb4b7f3c62421bab2d17f90f254e0416de31a"}
Sep 08 10:14:41.407917 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.407898    2559 scope.go:117] "RemoveContainer" containerID="b440511241c14b3befa9b8b22908ebd125dcf0773f3b7d32bee46c815e9258cd"
Sep 08 10:14:41.408206 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:14:41.408186    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b440511241c14b3befa9b8b22908ebd125dcf0773f3b7d32bee46c815e9258cd\": container with ID starting with b440511241c14b3befa9b8b22908ebd125dcf0773f3b7d32bee46c815e9258cd not found: ID does not exist" containerID="b440511241c14b3befa9b8b22908ebd125dcf0773f3b7d32bee46c815e9258cd"
Sep 08 10:14:41.408409 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.408218    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b440511241c14b3befa9b8b22908ebd125dcf0773f3b7d32bee46c815e9258cd"} err="failed to get container status \"b440511241c14b3befa9b8b22908ebd125dcf0773f3b7d32bee46c815e9258cd\": rpc error: code = NotFound desc = could not find container \"b440511241c14b3befa9b8b22908ebd125dcf0773f3b7d32bee46c815e9258cd\": container with ID starting with b440511241c14b3befa9b8b22908ebd125dcf0773f3b7d32bee46c815e9258cd not found: ID does not exist"
Sep 08 10:14:41.408409 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.408269    2559 scope.go:117] "RemoveContainer" containerID="396505c40bd5ee62a63b4847c217b978842ea552e8ec9312daab9683bed90e26"
Sep 08 10:14:41.416450 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.416431    2559 scope.go:117] "RemoveContainer" containerID="396505c40bd5ee62a63b4847c217b978842ea552e8ec9312daab9683bed90e26"
Sep 08 10:14:41.416737 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:14:41.416717    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396505c40bd5ee62a63b4847c217b978842ea552e8ec9312daab9683bed90e26\": container with ID starting with 396505c40bd5ee62a63b4847c217b978842ea552e8ec9312daab9683bed90e26 not found: ID does not exist" containerID="396505c40bd5ee62a63b4847c217b978842ea552e8ec9312daab9683bed90e26"
Sep 08 10:14:41.416810 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.416746    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396505c40bd5ee62a63b4847c217b978842ea552e8ec9312daab9683bed90e26"} err="failed to get container status \"396505c40bd5ee62a63b4847c217b978842ea552e8ec9312daab9683bed90e26\": rpc error: code = NotFound desc = could not find container \"396505c40bd5ee62a63b4847c217b978842ea552e8ec9312daab9683bed90e26\": container with ID starting with 396505c40bd5ee62a63b4847c217b978842ea552e8ec9312daab9683bed90e26 not found: ID does not exist"
Sep 08 10:14:41.446747 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.446725    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"]
Sep 08 10:14:41.452139 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.452119    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbf4760c-957d9567c-cs9v7"]
Sep 08 10:14:41.495462 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.495431    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"]
Sep 08 10:14:41.515655 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:41.515633    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbf4760c-7c777c957d-b9w6l"]
Sep 08 10:14:42.638549 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:42.638520    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2faee22b-8ac7-484e-8d17-df5d0d5b416b" path="/var/lib/kubelet/pods/2faee22b-8ac7-484e-8d17-df5d0d5b416b/volumes"
Sep 08 10:14:42.638907 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:42.638861    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="458798e1-ac60-463a-88d7-db289022248e" path="/var/lib/kubelet/pods/458798e1-ac60-463a-88d7-db289022248e/volumes"
Sep 08 10:14:42.639138 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:42.639126    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66e5c2a2-f076-438c-8120-255d5efb750a" path="/var/lib/kubelet/pods/66e5c2a2-f076-438c-8120-255d5efb750a/volumes"
Sep 08 10:14:45.134518 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.134444    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"]
Sep 08 10:14:45.134896 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.134881    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="458798e1-ac60-463a-88d7-db289022248e" containerName="payload-pre-processing"
Sep 08 10:14:45.134896 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.134892    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="458798e1-ac60-463a-88d7-db289022248e" containerName="payload-pre-processing"
Sep 08 10:14:45.134976 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.134905    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66e5c2a2-f076-438c-8120-255d5efb750a" containerName="revoke-keys"
Sep 08 10:14:45.134976 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.134911    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="66e5c2a2-f076-438c-8120-255d5efb750a" containerName="revoke-keys"
Sep 08 10:14:45.134976 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.134934    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2faee22b-8ac7-484e-8d17-df5d0d5b416b" containerName="payload-processing"
Sep 08 10:14:45.134976 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.134940    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2faee22b-8ac7-484e-8d17-df5d0d5b416b" containerName="payload-processing"
Sep 08 10:14:45.135113 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.135001    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="66e5c2a2-f076-438c-8120-255d5efb750a" containerName="revoke-keys"
Sep 08 10:14:45.135113 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.135009    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="458798e1-ac60-463a-88d7-db289022248e" containerName="payload-pre-processing"
Sep 08 10:14:45.135113 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.135014    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2faee22b-8ac7-484e-8d17-df5d0d5b416b" containerName="payload-processing"
Sep 08 10:14:45.142441 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.142417    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"
Sep 08 10:14:45.145695 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.145676    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a5df2a72\""
Sep 08 10:14:45.145768 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.145701    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a5df2a72-dockercfg-pspzf\""
Sep 08 10:14:45.151577 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.151551    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"]
Sep 08 10:14:45.242416 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.242377    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbf9b\" (UniqueName: \"kubernetes.io/projected/3ba42a48-37c4-42dc-b154-44d5e9c718f9-kube-api-access-jbf9b\") pod \"payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp\" (UID: \"3ba42a48-37c4-42dc-b154-44d5e9c718f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"
Sep 08 10:14:45.242536 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.242510    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba42a48-37c4-42dc-b154-44d5e9c718f9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp\" (UID: \"3ba42a48-37c4-42dc-b154-44d5e9c718f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"
Sep 08 10:14:45.343540 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.343507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbf9b\" (UniqueName: \"kubernetes.io/projected/3ba42a48-37c4-42dc-b154-44d5e9c718f9-kube-api-access-jbf9b\") pod \"payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp\" (UID: \"3ba42a48-37c4-42dc-b154-44d5e9c718f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"
Sep 08 10:14:45.343687 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.343650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba42a48-37c4-42dc-b154-44d5e9c718f9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp\" (UID: \"3ba42a48-37c4-42dc-b154-44d5e9c718f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"
Sep 08 10:14:45.344347 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.344326    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba42a48-37c4-42dc-b154-44d5e9c718f9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp\" (UID: \"3ba42a48-37c4-42dc-b154-44d5e9c718f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"
Sep 08 10:14:45.351430 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.351406    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbf9b\" (UniqueName: \"kubernetes.io/projected/3ba42a48-37c4-42dc-b154-44d5e9c718f9-kube-api-access-jbf9b\") pod \"payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp\" (UID: \"3ba42a48-37c4-42dc-b154-44d5e9c718f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"
Sep 08 10:14:45.453999 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.453971    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"
Sep 08 10:14:45.727676 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:45.727654    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"]
Sep 08 10:14:45.729770 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:14:45.729746    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba42a48_37c4_42dc_b154_44d5e9c718f9.slice/crio-4759011057f85659ca6058bbd4ed0da93bbf7cc61903f10ed3b127b3e1735012 WatchSource:0}: Error finding container 4759011057f85659ca6058bbd4ed0da93bbf7cc61903f10ed3b127b3e1735012: Status 404 returned error can't find the container with id 4759011057f85659ca6058bbd4ed0da93bbf7cc61903f10ed3b127b3e1735012
Sep 08 10:14:46.431187 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:46.431155    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp" event={"ID":"3ba42a48-37c4-42dc-b154-44d5e9c718f9","Type":"ContainerStarted","Data":"16aa3a03105b38f619f15949fa4e9333e20644419b9d8eb01123b6e075ab97c1"}
Sep 08 10:14:46.431638 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:46.431203    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp" event={"ID":"3ba42a48-37c4-42dc-b154-44d5e9c718f9","Type":"ContainerStarted","Data":"4759011057f85659ca6058bbd4ed0da93bbf7cc61903f10ed3b127b3e1735012"}
Sep 08 10:14:46.431638 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:46.431295    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"
Sep 08 10:14:46.459686 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:46.459639    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp" podStartSLOduration=1.4596271 podStartE2EDuration="1.4596271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:14:46.457898593 +0000 UTC m=+806.299122998" watchObservedRunningTime="2026-09-08 10:14:46.4596271 +0000 UTC m=+806.300851483"
Sep 08 10:14:57.439336 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:14:57.439298    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"
Sep 08 10:15:00.148862 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.148821    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-338aba-29814375-95zwp"]
Sep 08 10:15:00.153000 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.152980    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814375-lzhph"]
Sep 08 10:15:00.153132 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.153089    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-338aba-29814375-95zwp"
Sep 08 10:15:00.155400 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.155374    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:15:00.155512 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.155404    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e365b-29814375-9kjsf"]
Sep 08 10:15:00.155512 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.155496    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814375-lzhph"
Sep 08 10:15:00.158210 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.158188    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e365b-29814375-9kjsf"
Sep 08 10:15:00.159944 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.159924    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e365b-29814375-9kjsf"]
Sep 08 10:15:00.162937 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.162480    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814375-lzhph"]
Sep 08 10:15:00.164993 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.164971    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-338aba-29814375-95zwp"]
Sep 08 10:15:00.166622 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.166605    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-338aba-29814375-95zwp"
Sep 08 10:15:00.198384 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.198361    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814375-lzhph"
Sep 08 10:15:00.205524 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.205504    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e365b-29814375-9kjsf"
Sep 08 10:15:00.462863 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.462841    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-338aba-29814375-95zwp"]
Sep 08 10:15:00.464547 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:15:00.464516    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a22068f_2b6b_4cf9_bc67_6ecdd242f7aa.slice/crio-c49d47ea6ec67231c4afe0f1f11739940ae845d50faccc914d2203a24a1102c2 WatchSource:0}: Error finding container c49d47ea6ec67231c4afe0f1f11739940ae845d50faccc914d2203a24a1102c2: Status 404 returned error can't find the container with id c49d47ea6ec67231c4afe0f1f11739940ae845d50faccc914d2203a24a1102c2
Sep 08 10:15:00.493623 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.493604    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814375-lzhph"]
Sep 08 10:15:00.495105 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:15:00.495075    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5bedf1_9180_49a3_b5ed_5471893ebc1f.slice/crio-85e4b9fd704869fe2da5ef47606861b306ecea81ea0eb1de80d2e36bcc63c447 WatchSource:0}: Error finding container 85e4b9fd704869fe2da5ef47606861b306ecea81ea0eb1de80d2e36bcc63c447: Status 404 returned error can't find the container with id 85e4b9fd704869fe2da5ef47606861b306ecea81ea0eb1de80d2e36bcc63c447
Sep 08 10:15:00.505858 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.505776    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e365b-29814375-9kjsf"]
Sep 08 10:15:00.508267 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.507911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-338aba-29814375-95zwp" event={"ID":"6a22068f-2b6b-4cf9-bc67-6ecdd242f7aa","Type":"ContainerStarted","Data":"c49d47ea6ec67231c4afe0f1f11739940ae845d50faccc914d2203a24a1102c2"}
Sep 08 10:15:00.508267 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:15:00.508018    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5bda56_cf56_4f6d_8b35_1086108d7503.slice/crio-96396bae3d07c665aba43823f9d15c41c067801bf751fd762a923c8b9e79d637 WatchSource:0}: Error finding container 96396bae3d07c665aba43823f9d15c41c067801bf751fd762a923c8b9e79d637: Status 404 returned error can't find the container with id 96396bae3d07c665aba43823f9d15c41c067801bf751fd762a923c8b9e79d637
Sep 08 10:15:00.509944 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:00.509897    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814375-lzhph" event={"ID":"ad5bedf1-9180-49a3-b5ed-5471893ebc1f","Type":"ContainerStarted","Data":"85e4b9fd704869fe2da5ef47606861b306ecea81ea0eb1de80d2e36bcc63c447"}
Sep 08 10:15:01.516951 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:01.516920    2559 generic.go:356] "Generic (PLEG): container finished" podID="cc5bda56-cf56-4f6d-8b35-1086108d7503" containerID="6b930fa8e5627491fb609bd8e5a4026e929c72515aee92789e0f1da84b0912a9" exitCode=0
Sep 08 10:15:01.517372 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:01.516991    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e365b-29814375-9kjsf" event={"ID":"cc5bda56-cf56-4f6d-8b35-1086108d7503","Type":"ContainerDied","Data":"6b930fa8e5627491fb609bd8e5a4026e929c72515aee92789e0f1da84b0912a9"}
Sep 08 10:15:01.517372 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:01.517036    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e365b-29814375-9kjsf" event={"ID":"cc5bda56-cf56-4f6d-8b35-1086108d7503","Type":"ContainerStarted","Data":"96396bae3d07c665aba43823f9d15c41c067801bf751fd762a923c8b9e79d637"}
Sep 08 10:15:01.518520 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:01.518496    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-338aba-29814375-95zwp" event={"ID":"6a22068f-2b6b-4cf9-bc67-6ecdd242f7aa","Type":"ContainerStarted","Data":"b8e877b9ca76594a6ff2f01b92eb2f045e7dcdac4bcbe28abf7614e13fb80872"}
Sep 08 10:15:01.520102 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:01.520081    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814375-lzhph" event={"ID":"ad5bedf1-9180-49a3-b5ed-5471893ebc1f","Type":"ContainerStarted","Data":"e5b68e4a8305b4ac02d805be51ffa58ce48537e855311d41ce5b7a73723cfb63"}
Sep 08 10:15:01.561431 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:01.561391    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-338aba-29814375-95zwp" podStartSLOduration=1.561377287 podStartE2EDuration="1.561377287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:15:01.559359815 +0000 UTC m=+821.400584223" watchObservedRunningTime="2026-09-08 10:15:01.561377287 +0000 UTC m=+821.402601661"
Sep 08 10:15:01.574102 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:01.574065    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814375-lzhph" podStartSLOduration=1.5740541129999999 podStartE2EDuration="1.574054113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:15:01.574036247 +0000 UTC m=+821.415260629" watchObservedRunningTime="2026-09-08 10:15:01.574054113 +0000 UTC m=+821.415278610"
Sep 08 10:15:02.526983 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:02.526946    2559 generic.go:356] "Generic (PLEG): container finished" podID="ad5bedf1-9180-49a3-b5ed-5471893ebc1f" containerID="e5b68e4a8305b4ac02d805be51ffa58ce48537e855311d41ce5b7a73723cfb63" exitCode=0
Sep 08 10:15:02.527408 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:02.527025    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814375-lzhph" event={"ID":"ad5bedf1-9180-49a3-b5ed-5471893ebc1f","Type":"ContainerDied","Data":"e5b68e4a8305b4ac02d805be51ffa58ce48537e855311d41ce5b7a73723cfb63"}
Sep 08 10:15:02.528617 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:02.528591    2559 generic.go:356] "Generic (PLEG): container finished" podID="6a22068f-2b6b-4cf9-bc67-6ecdd242f7aa" containerID="b8e877b9ca76594a6ff2f01b92eb2f045e7dcdac4bcbe28abf7614e13fb80872" exitCode=0
Sep 08 10:15:02.528731 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:02.528645    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-338aba-29814375-95zwp" event={"ID":"6a22068f-2b6b-4cf9-bc67-6ecdd242f7aa","Type":"ContainerDied","Data":"b8e877b9ca76594a6ff2f01b92eb2f045e7dcdac4bcbe28abf7614e13fb80872"}
Sep 08 10:15:02.674770 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:02.674747    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e365b-29814375-9kjsf"
Sep 08 10:15:03.535786 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:03.535738    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e365b-29814375-9kjsf" event={"ID":"cc5bda56-cf56-4f6d-8b35-1086108d7503","Type":"ContainerDied","Data":"96396bae3d07c665aba43823f9d15c41c067801bf751fd762a923c8b9e79d637"}
Sep 08 10:15:03.535786 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:03.535762    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e365b-29814375-9kjsf"
Sep 08 10:15:03.536362 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:03.535801    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96396bae3d07c665aba43823f9d15c41c067801bf751fd762a923c8b9e79d637"
Sep 08 10:15:03.695930 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:03.695911    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814375-lzhph"
Sep 08 10:15:03.698943 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:03.698923    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-338aba-29814375-95zwp"
Sep 08 10:15:04.543002 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:04.542974    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-338aba-29814375-95zwp"
Sep 08 10:15:04.543545 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:04.542968    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-338aba-29814375-95zwp" event={"ID":"6a22068f-2b6b-4cf9-bc67-6ecdd242f7aa","Type":"ContainerDied","Data":"c49d47ea6ec67231c4afe0f1f11739940ae845d50faccc914d2203a24a1102c2"}
Sep 08 10:15:04.543545 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:04.543111    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c49d47ea6ec67231c4afe0f1f11739940ae845d50faccc914d2203a24a1102c2"
Sep 08 10:15:04.544499 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:04.544476    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814375-lzhph" event={"ID":"ad5bedf1-9180-49a3-b5ed-5471893ebc1f","Type":"ContainerDied","Data":"85e4b9fd704869fe2da5ef47606861b306ecea81ea0eb1de80d2e36bcc63c447"}
Sep 08 10:15:04.544637 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:04.544503    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814375-lzhph"
Sep 08 10:15:04.544637 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:04.544512    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85e4b9fd704869fe2da5ef47606861b306ecea81ea0eb1de80d2e36bcc63c447"
Sep 08 10:15:09.278542 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.278507    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf"]
Sep 08 10:15:09.279119 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.279099    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a22068f-2b6b-4cf9-bc67-6ecdd242f7aa" containerName="cleanup"
Sep 08 10:15:09.279204 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.279122    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a22068f-2b6b-4cf9-bc67-6ecdd242f7aa" containerName="cleanup"
Sep 08 10:15:09.279204 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.279136    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5bedf1-9180-49a3-b5ed-5471893ebc1f" containerName="cleanup"
Sep 08 10:15:09.279204 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.279144    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5bedf1-9180-49a3-b5ed-5471893ebc1f" containerName="cleanup"
Sep 08 10:15:09.279204 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.279178    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc5bda56-cf56-4f6d-8b35-1086108d7503" containerName="cleanup"
Sep 08 10:15:09.279204 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.279186    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc5bda56-cf56-4f6d-8b35-1086108d7503" containerName="cleanup"
Sep 08 10:15:09.279491 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.279356    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc5bda56-cf56-4f6d-8b35-1086108d7503" containerName="cleanup"
Sep 08 10:15:09.279491 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.279376    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5bedf1-9180-49a3-b5ed-5471893ebc1f" containerName="cleanup"
Sep 08 10:15:09.279491 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.279387    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a22068f-2b6b-4cf9-bc67-6ecdd242f7aa" containerName="cleanup"
Sep 08 10:15:09.282660 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.282640    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf"
Sep 08 10:15:09.285316 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.285294    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:15:09.285431 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.285372    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:15:09.292013 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.291988    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf"]
Sep 08 10:15:09.359642 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.359611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38743adb-598e-4a3f-8d2c-8d11385ce2cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf\" (UID: \"38743adb-598e-4a3f-8d2c-8d11385ce2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf"
Sep 08 10:15:09.460523 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.460484    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38743adb-598e-4a3f-8d2c-8d11385ce2cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf\" (UID: \"38743adb-598e-4a3f-8d2c-8d11385ce2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf"
Sep 08 10:15:09.461122 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.461099    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38743adb-598e-4a3f-8d2c-8d11385ce2cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf\" (UID: \"38743adb-598e-4a3f-8d2c-8d11385ce2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf"
Sep 08 10:15:09.594882 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.594803    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf"
Sep 08 10:15:09.867144 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:09.867107    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf"]
Sep 08 10:15:09.868697 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:15:09.868669    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38743adb_598e_4a3f_8d2c_8d11385ce2cc.slice/crio-a115a42d4fcac82827ec805539a6abe0e9362aa083318a4fd74e5287f915610b WatchSource:0}: Error finding container a115a42d4fcac82827ec805539a6abe0e9362aa083318a4fd74e5287f915610b: Status 404 returned error can't find the container with id a115a42d4fcac82827ec805539a6abe0e9362aa083318a4fd74e5287f915610b
Sep 08 10:15:10.579869 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:10.579840    2559 generic.go:356] "Generic (PLEG): container finished" podID="38743adb-598e-4a3f-8d2c-8d11385ce2cc" containerID="71fbf80a789d2932de835544fa61eae96c56af27f41e77946a4f621a8dc51743" exitCode=0
Sep 08 10:15:10.580236 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:10.579883    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf" event={"ID":"38743adb-598e-4a3f-8d2c-8d11385ce2cc","Type":"ContainerDied","Data":"71fbf80a789d2932de835544fa61eae96c56af27f41e77946a4f621a8dc51743"}
Sep 08 10:15:10.580236 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:10.579917    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf" event={"ID":"38743adb-598e-4a3f-8d2c-8d11385ce2cc","Type":"ContainerStarted","Data":"a115a42d4fcac82827ec805539a6abe0e9362aa083318a4fd74e5287f915610b"}
Sep 08 10:15:11.725111 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:11.725085    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf"
Sep 08 10:15:11.780446 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:11.780420    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38743adb-598e-4a3f-8d2c-8d11385ce2cc-maas-api-service-ca\") pod \"38743adb-598e-4a3f-8d2c-8d11385ce2cc\" (UID: \"38743adb-598e-4a3f-8d2c-8d11385ce2cc\") "
Sep 08 10:15:11.780740 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:11.780718    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38743adb-598e-4a3f-8d2c-8d11385ce2cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38743adb-598e-4a3f-8d2c-8d11385ce2cc" (UID: "38743adb-598e-4a3f-8d2c-8d11385ce2cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:15:11.881381 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:11.881325    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38743adb-598e-4a3f-8d2c-8d11385ce2cc-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:15:12.592553 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:12.592521    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf"
Sep 08 10:15:12.592740 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:12.592572    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf" event={"ID":"38743adb-598e-4a3f-8d2c-8d11385ce2cc","Type":"ContainerDied","Data":"a115a42d4fcac82827ec805539a6abe0e9362aa083318a4fd74e5287f915610b"}
Sep 08 10:15:12.592740 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:12.592659    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a115a42d4fcac82827ec805539a6abe0e9362aa083318a4fd74e5287f915610b"
Sep 08 10:15:16.035681 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.035649    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"]
Sep 08 10:15:16.036418 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.036382    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp" podUID="3ba42a48-37c4-42dc-b154-44d5e9c718f9" containerName="payload-processing" containerID="cri-o://16aa3a03105b38f619f15949fa4e9333e20644419b9d8eb01123b6e075ab97c1" gracePeriod=30
Sep 08 10:15:16.226269 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.223337    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf"]
Sep 08 10:15:16.230394 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.230367    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5df2a72-ldwqf"]
Sep 08 10:15:16.310082 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.310060    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"
Sep 08 10:15:16.422297 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.422259    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbf9b\" (UniqueName: \"kubernetes.io/projected/3ba42a48-37c4-42dc-b154-44d5e9c718f9-kube-api-access-jbf9b\") pod \"3ba42a48-37c4-42dc-b154-44d5e9c718f9\" (UID: \"3ba42a48-37c4-42dc-b154-44d5e9c718f9\") "
Sep 08 10:15:16.422475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.422327    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba42a48-37c4-42dc-b154-44d5e9c718f9-plugins-config-volume\") pod \"3ba42a48-37c4-42dc-b154-44d5e9c718f9\" (UID: \"3ba42a48-37c4-42dc-b154-44d5e9c718f9\") "
Sep 08 10:15:16.422680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.422658    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba42a48-37c4-42dc-b154-44d5e9c718f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ba42a48-37c4-42dc-b154-44d5e9c718f9" (UID: "3ba42a48-37c4-42dc-b154-44d5e9c718f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:15:16.424238 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.424214    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba42a48-37c4-42dc-b154-44d5e9c718f9-kube-api-access-jbf9b" (OuterVolumeSpecName: "kube-api-access-jbf9b") pod "3ba42a48-37c4-42dc-b154-44d5e9c718f9" (UID: "3ba42a48-37c4-42dc-b154-44d5e9c718f9"). InnerVolumeSpecName "kube-api-access-jbf9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:15:16.523005 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.522976    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba42a48-37c4-42dc-b154-44d5e9c718f9-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:15:16.523005 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.523003    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbf9b\" (UniqueName: \"kubernetes.io/projected/3ba42a48-37c4-42dc-b154-44d5e9c718f9-kube-api-access-jbf9b\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:15:16.615163 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.615083    2559 generic.go:356] "Generic (PLEG): container finished" podID="3ba42a48-37c4-42dc-b154-44d5e9c718f9" containerID="16aa3a03105b38f619f15949fa4e9333e20644419b9d8eb01123b6e075ab97c1" exitCode=0
Sep 08 10:15:16.615163 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.615128    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"
Sep 08 10:15:16.615361 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.615173    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp" event={"ID":"3ba42a48-37c4-42dc-b154-44d5e9c718f9","Type":"ContainerDied","Data":"16aa3a03105b38f619f15949fa4e9333e20644419b9d8eb01123b6e075ab97c1"}
Sep 08 10:15:16.615361 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.615276    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp" event={"ID":"3ba42a48-37c4-42dc-b154-44d5e9c718f9","Type":"ContainerDied","Data":"4759011057f85659ca6058bbd4ed0da93bbf7cc61903f10ed3b127b3e1735012"}
Sep 08 10:15:16.615361 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.615308    2559 scope.go:117] "RemoveContainer" containerID="16aa3a03105b38f619f15949fa4e9333e20644419b9d8eb01123b6e075ab97c1"
Sep 08 10:15:16.624364 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.624330    2559 scope.go:117] "RemoveContainer" containerID="16aa3a03105b38f619f15949fa4e9333e20644419b9d8eb01123b6e075ab97c1"
Sep 08 10:15:16.624623 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:15:16.624602    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16aa3a03105b38f619f15949fa4e9333e20644419b9d8eb01123b6e075ab97c1\": container with ID starting with 16aa3a03105b38f619f15949fa4e9333e20644419b9d8eb01123b6e075ab97c1 not found: ID does not exist" containerID="16aa3a03105b38f619f15949fa4e9333e20644419b9d8eb01123b6e075ab97c1"
Sep 08 10:15:16.624714 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.624629    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16aa3a03105b38f619f15949fa4e9333e20644419b9d8eb01123b6e075ab97c1"} err="failed to get container status \"16aa3a03105b38f619f15949fa4e9333e20644419b9d8eb01123b6e075ab97c1\": rpc error: code = NotFound desc = could not find container \"16aa3a03105b38f619f15949fa4e9333e20644419b9d8eb01123b6e075ab97c1\": container with ID starting with 16aa3a03105b38f619f15949fa4e9333e20644419b9d8eb01123b6e075ab97c1 not found: ID does not exist"
Sep 08 10:15:16.636766 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.636742    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38743adb-598e-4a3f-8d2c-8d11385ce2cc" path="/var/lib/kubelet/pods/38743adb-598e-4a3f-8d2c-8d11385ce2cc/volumes"
Sep 08 10:15:16.638055 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.638037    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"]
Sep 08 10:15:16.644628 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:16.644606    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a5df2a72-cbc8ddcd-frphp"]
Sep 08 10:15:18.636868 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:18.636836    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ba42a48-37c4-42dc-b154-44d5e9c718f9" path="/var/lib/kubelet/pods/3ba42a48-37c4-42dc-b154-44d5e9c718f9/volumes"
Sep 08 10:15:32.941960 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:32.941919    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"]
Sep 08 10:15:32.942640 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:32.942618    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba42a48-37c4-42dc-b154-44d5e9c718f9" containerName="payload-processing"
Sep 08 10:15:32.942686 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:32.942646    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba42a48-37c4-42dc-b154-44d5e9c718f9" containerName="payload-processing"
Sep 08 10:15:32.942724 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:32.942696    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38743adb-598e-4a3f-8d2c-8d11385ce2cc" containerName="revoke-keys"
Sep 08 10:15:32.942724 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:32.942706    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="38743adb-598e-4a3f-8d2c-8d11385ce2cc" containerName="revoke-keys"
Sep 08 10:15:32.942846 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:32.942832    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="38743adb-598e-4a3f-8d2c-8d11385ce2cc" containerName="revoke-keys"
Sep 08 10:15:32.942879 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:32.942853    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba42a48-37c4-42dc-b154-44d5e9c718f9" containerName="payload-processing"
Sep 08 10:15:32.947907 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:32.947885    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:32.953813 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:32.953789    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-338aba\"/\"default-dockercfg-pd56c\""
Sep 08 10:15:32.954005 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:32.953859    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-338aba\"/\"openshift-service-ca.crt\""
Sep 08 10:15:32.954005 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:32.953838    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-338aba\"/\"kube-root-ca.crt\""
Sep 08 10:15:32.954005 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:32.953829    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-338aba\"/\"autores-model-338aba-kserve-self-signed-certs\""
Sep 08 10:15:32.958184 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:32.958157    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"]
Sep 08 10:15:33.068048 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.068018    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae079d35-f357-4b67-b143-d631b6cfe44d-tls-certs\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.068215 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.068062    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-home\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.068215 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.068105    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-model-cache\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.068215 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.068186    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-tmp-dir\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.068457 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.068238    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-dshm\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.068457 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.068330    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7mzh\" (UniqueName: \"kubernetes.io/projected/ae079d35-f357-4b67-b143-d631b6cfe44d-kube-api-access-j7mzh\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.168842 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.168817    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-dshm\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.169004 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.168859    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7mzh\" (UniqueName: \"kubernetes.io/projected/ae079d35-f357-4b67-b143-d631b6cfe44d-kube-api-access-j7mzh\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.169004 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.168882    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae079d35-f357-4b67-b143-d631b6cfe44d-tls-certs\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.169119 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.168998    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-home\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.169119 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.169044    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-model-cache\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.169119 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.169085    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-tmp-dir\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.169396 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.169366    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-home\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.169529 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.169410    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-model-cache\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.169529 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.169441    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-tmp-dir\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.171049 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.171023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-dshm\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.171347 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.171330    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae079d35-f357-4b67-b143-d631b6cfe44d-tls-certs\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.178355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.178328    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7mzh\" (UniqueName: \"kubernetes.io/projected/ae079d35-f357-4b67-b143-d631b6cfe44d-kube-api-access-j7mzh\") pod \"autores-model-338aba-kserve-6f4c95d5c9-z9zpd\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") " pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.259824 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.259769    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:33.532553 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.532521    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"]
Sep 08 10:15:33.534340 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:15:33.534309    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae079d35_f357_4b67_b143_d631b6cfe44d.slice/crio-0946f525f5adbee828a65a20cac68b579de06b2335321f6314773dc0fa5637d8 WatchSource:0}: Error finding container 0946f525f5adbee828a65a20cac68b579de06b2335321f6314773dc0fa5637d8: Status 404 returned error can't find the container with id 0946f525f5adbee828a65a20cac68b579de06b2335321f6314773dc0fa5637d8
Sep 08 10:15:33.708221 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.708183    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd" event={"ID":"ae079d35-f357-4b67-b143-d631b6cfe44d","Type":"ContainerStarted","Data":"06514c28a317604873cfea12b739f375f2f1c519e4cd08b01340d44d14a16f48"}
Sep 08 10:15:33.708221 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.708233    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd" event={"ID":"ae079d35-f357-4b67-b143-d631b6cfe44d","Type":"ContainerStarted","Data":"0946f525f5adbee828a65a20cac68b579de06b2335321f6314773dc0fa5637d8"}
Sep 08 10:15:33.732412 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:33.732373    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd" podStartSLOduration=1.7323596669999999 podStartE2EDuration="1.732359667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:15:33.731213063 +0000 UTC m=+853.572437447" watchObservedRunningTime="2026-09-08 10:15:33.732359667 +0000 UTC m=+853.573584085"
Sep 08 10:15:43.260256 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:43.260212    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:43.260735 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:43.260273    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:43.273458 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:43.273434    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:43.770074 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:43.770047    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:15:45.598995 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:45.597024    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc"]
Sep 08 10:15:45.603729 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:45.603703    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc"
Sep 08 10:15:45.606781 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:45.606754    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:15:45.606781 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:45.606767    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:15:45.608555 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:45.608530    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc"]
Sep 08 10:15:45.691085 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:45.691059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/519cd2d3-dbbf-4145-910d-ca32c0f2eab4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc\" (UID: \"519cd2d3-dbbf-4145-910d-ca32c0f2eab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc"
Sep 08 10:15:45.792043 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:45.792021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/519cd2d3-dbbf-4145-910d-ca32c0f2eab4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc\" (UID: \"519cd2d3-dbbf-4145-910d-ca32c0f2eab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc"
Sep 08 10:15:45.792597 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:45.792578    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/519cd2d3-dbbf-4145-910d-ca32c0f2eab4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc\" (UID: \"519cd2d3-dbbf-4145-910d-ca32c0f2eab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc"
Sep 08 10:15:45.916281 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:45.916256    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc"
Sep 08 10:15:46.190935 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:46.190913    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc"]
Sep 08 10:15:46.192885 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:15:46.192858    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519cd2d3_dbbf_4145_910d_ca32c0f2eab4.slice/crio-ddbdf0c662eaf75aefc0ab4ceecab682f75b33382270b8016102e64526304b3f WatchSource:0}: Error finding container ddbdf0c662eaf75aefc0ab4ceecab682f75b33382270b8016102e64526304b3f: Status 404 returned error can't find the container with id ddbdf0c662eaf75aefc0ab4ceecab682f75b33382270b8016102e64526304b3f
Sep 08 10:15:46.776030 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:46.775941    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc" event={"ID":"519cd2d3-dbbf-4145-910d-ca32c0f2eab4","Type":"ContainerStarted","Data":"c976a505884d76428bd0c875f6a4c002f4157b12ad7ff205fb81c0d7c75691b6"}
Sep 08 10:15:46.776030 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:46.775998    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc" event={"ID":"519cd2d3-dbbf-4145-910d-ca32c0f2eab4","Type":"ContainerStarted","Data":"ddbdf0c662eaf75aefc0ab4ceecab682f75b33382270b8016102e64526304b3f"}
Sep 08 10:15:46.807402 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:46.807354    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc" podStartSLOduration=1.8073402920000001 podStartE2EDuration="1.807340292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:15:46.805690915 +0000 UTC m=+866.646915297" watchObservedRunningTime="2026-09-08 10:15:46.807340292 +0000 UTC m=+866.648564671"
Sep 08 10:15:47.782881 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:47.782852    2559 generic.go:356] "Generic (PLEG): container finished" podID="519cd2d3-dbbf-4145-910d-ca32c0f2eab4" containerID="c976a505884d76428bd0c875f6a4c002f4157b12ad7ff205fb81c0d7c75691b6" exitCode=0
Sep 08 10:15:47.783271 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:47.782906    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc" event={"ID":"519cd2d3-dbbf-4145-910d-ca32c0f2eab4","Type":"ContainerDied","Data":"c976a505884d76428bd0c875f6a4c002f4157b12ad7ff205fb81c0d7c75691b6"}
Sep 08 10:15:48.110411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.110331    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"]
Sep 08 10:15:48.113837 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.113816    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.120371 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.120299    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c28013\"/\"kube-root-ca.crt\""
Sep 08 10:15:48.121215 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.121196    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c28013\"/\"openshift-service-ca.crt\""
Sep 08 10:15:48.121345 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.121235    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c28013\"/\"default-dockercfg-sxwh6\""
Sep 08 10:15:48.121345 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.121289    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c28013\"/\"test-model-c28013-kserve-self-signed-certs\""
Sep 08 10:15:48.127560 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.127541    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"]
Sep 08 10:15:48.214398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.214361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-dshm\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.214529 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.214416    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-model-cache\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.214529 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.214437    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-home\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.214529 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.214463    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46dbde84-2663-4cac-9d32-9627cdbd8288-tls-certs\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.214529 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.214500    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-tmp-dir\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.214529 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.214521    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg2fd\" (UniqueName: \"kubernetes.io/projected/46dbde84-2663-4cac-9d32-9627cdbd8288-kube-api-access-xg2fd\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.315214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.315186    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-dshm\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.315355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.315233    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-model-cache\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.315355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.315289    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-home\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.315355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.315307    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46dbde84-2663-4cac-9d32-9627cdbd8288-tls-certs\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.315355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.315326    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-tmp-dir\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.315542 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.315357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg2fd\" (UniqueName: \"kubernetes.io/projected/46dbde84-2663-4cac-9d32-9627cdbd8288-kube-api-access-xg2fd\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.315672 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.315648    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-model-cache\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.315738 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.315702    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-home\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.315794 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.315744    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-tmp-dir\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.317411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.317389    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-dshm\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.317787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.317768    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46dbde84-2663-4cac-9d32-9627cdbd8288-tls-certs\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.323768 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.323748    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg2fd\" (UniqueName: \"kubernetes.io/projected/46dbde84-2663-4cac-9d32-9627cdbd8288-kube-api-access-xg2fd\") pod \"test-model-c28013-kserve-7df7c86955-kkdqn\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") " pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.422735 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.422710    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:48.696746 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.696722    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"]
Sep 08 10:15:48.698156 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:15:48.698117    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46dbde84_2663_4cac_9d32_9627cdbd8288.slice/crio-e41b080dcd460267d75c253e21d936922c840a813576a33986af532b734c4467 WatchSource:0}: Error finding container e41b080dcd460267d75c253e21d936922c840a813576a33986af532b734c4467: Status 404 returned error can't find the container with id e41b080dcd460267d75c253e21d936922c840a813576a33986af532b734c4467
Sep 08 10:15:48.790674 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.790640    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn" event={"ID":"46dbde84-2663-4cac-9d32-9627cdbd8288","Type":"ContainerStarted","Data":"68652c866b16351498fdd60af06216a2a98974e632786a23fb3654b75ec6c224"}
Sep 08 10:15:48.791014 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.790685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn" event={"ID":"46dbde84-2663-4cac-9d32-9627cdbd8288","Type":"ContainerStarted","Data":"e41b080dcd460267d75c253e21d936922c840a813576a33986af532b734c4467"}
Sep 08 10:15:48.815864 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.815811    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn" podStartSLOduration=0.815793806 podStartE2EDuration="815.793806ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:15:48.814965296 +0000 UTC m=+868.656189684" watchObservedRunningTime="2026-09-08 10:15:48.815793806 +0000 UTC m=+868.657018208"
Sep 08 10:15:48.915953 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:48.915928    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc"
Sep 08 10:15:49.021860 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:49.021791    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/519cd2d3-dbbf-4145-910d-ca32c0f2eab4-maas-api-service-ca\") pod \"519cd2d3-dbbf-4145-910d-ca32c0f2eab4\" (UID: \"519cd2d3-dbbf-4145-910d-ca32c0f2eab4\") "
Sep 08 10:15:49.022135 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:49.022114    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519cd2d3-dbbf-4145-910d-ca32c0f2eab4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "519cd2d3-dbbf-4145-910d-ca32c0f2eab4" (UID: "519cd2d3-dbbf-4145-910d-ca32c0f2eab4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:15:49.122970 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:49.122945    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/519cd2d3-dbbf-4145-910d-ca32c0f2eab4-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:15:49.797207 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:49.797181    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc"
Sep 08 10:15:49.797608 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:49.797215    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc" event={"ID":"519cd2d3-dbbf-4145-910d-ca32c0f2eab4","Type":"ContainerDied","Data":"ddbdf0c662eaf75aefc0ab4ceecab682f75b33382270b8016102e64526304b3f"}
Sep 08 10:15:49.797608 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:49.797275    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddbdf0c662eaf75aefc0ab4ceecab682f75b33382270b8016102e64526304b3f"
Sep 08 10:15:52.093214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:52.093176    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc"]
Sep 08 10:15:52.098613 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:52.098589    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f688fb08-nvkzc"]
Sep 08 10:15:52.637350 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:52.637316    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="519cd2d3-dbbf-4145-910d-ca32c0f2eab4" path="/var/lib/kubelet/pods/519cd2d3-dbbf-4145-910d-ca32c0f2eab4/volumes"
Sep 08 10:15:53.034770 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:53.034737    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"]
Sep 08 10:15:53.303001 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:53.302929    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55"]
Sep 08 10:15:53.303395 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:53.303381    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="519cd2d3-dbbf-4145-910d-ca32c0f2eab4" containerName="revoke-keys"
Sep 08 10:15:53.303395 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:53.303396    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="519cd2d3-dbbf-4145-910d-ca32c0f2eab4" containerName="revoke-keys"
Sep 08 10:15:53.303487 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:53.303457    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="519cd2d3-dbbf-4145-910d-ca32c0f2eab4" containerName="revoke-keys"
Sep 08 10:15:53.306697 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:53.306680    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55"
Sep 08 10:15:53.309190 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:53.309166    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:15:53.309393 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:53.309267    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:15:53.318216 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:53.318196    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55"]
Sep 08 10:15:53.359563 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:53.359529    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c7225c-329c-42ae-a8bc-ac7e4d6cf20e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-338aba-hsd55\" (UID: \"64c7225c-329c-42ae-a8bc-ac7e4d6cf20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55"
Sep 08 10:15:53.460924 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:53.460878    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c7225c-329c-42ae-a8bc-ac7e4d6cf20e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-338aba-hsd55\" (UID: \"64c7225c-329c-42ae-a8bc-ac7e4d6cf20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55"
Sep 08 10:15:53.461544 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:53.461526    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c7225c-329c-42ae-a8bc-ac7e4d6cf20e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-338aba-hsd55\" (UID: \"64c7225c-329c-42ae-a8bc-ac7e4d6cf20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55"
Sep 08 10:15:53.620134 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:53.620044    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55"
Sep 08 10:15:53.897064 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:53.897035    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55"]
Sep 08 10:15:53.898052 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:15:53.898025    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c7225c_329c_42ae_a8bc_ac7e4d6cf20e.slice/crio-69fcba791204710c87505284284f5586396cc887d6fe11eef0b949c894eaf09a WatchSource:0}: Error finding container 69fcba791204710c87505284284f5586396cc887d6fe11eef0b949c894eaf09a: Status 404 returned error can't find the container with id 69fcba791204710c87505284284f5586396cc887d6fe11eef0b949c894eaf09a
Sep 08 10:15:54.828556 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:54.828516    2559 generic.go:356] "Generic (PLEG): container finished" podID="64c7225c-329c-42ae-a8bc-ac7e4d6cf20e" containerID="cf6c2efb1d26af46cddf5b7276104ef726239b00da7dc242e4544f173fcf7613" exitCode=0
Sep 08 10:15:54.828556 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:54.828559    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55" event={"ID":"64c7225c-329c-42ae-a8bc-ac7e4d6cf20e","Type":"ContainerDied","Data":"cf6c2efb1d26af46cddf5b7276104ef726239b00da7dc242e4544f173fcf7613"}
Sep 08 10:15:54.829011 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:54.828595    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55" event={"ID":"64c7225c-329c-42ae-a8bc-ac7e4d6cf20e","Type":"ContainerStarted","Data":"69fcba791204710c87505284284f5586396cc887d6fe11eef0b949c894eaf09a"}
Sep 08 10:15:55.970869 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:55.970846    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55"
Sep 08 10:15:56.086687 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:56.086591    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c7225c-329c-42ae-a8bc-ac7e4d6cf20e-maas-api-service-ca\") pod \"64c7225c-329c-42ae-a8bc-ac7e4d6cf20e\" (UID: \"64c7225c-329c-42ae-a8bc-ac7e4d6cf20e\") "
Sep 08 10:15:56.086962 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:56.086936    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c7225c-329c-42ae-a8bc-ac7e4d6cf20e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64c7225c-329c-42ae-a8bc-ac7e4d6cf20e" (UID: "64c7225c-329c-42ae-a8bc-ac7e4d6cf20e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:15:56.187637 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:56.187603    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c7225c-329c-42ae-a8bc-ac7e4d6cf20e-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:15:56.840606 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:56.840574    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55"
Sep 08 10:15:56.840760 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:56.840612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55" event={"ID":"64c7225c-329c-42ae-a8bc-ac7e4d6cf20e","Type":"ContainerDied","Data":"69fcba791204710c87505284284f5586396cc887d6fe11eef0b949c894eaf09a"}
Sep 08 10:15:56.840760 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:56.840662    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69fcba791204710c87505284284f5586396cc887d6fe11eef0b949c894eaf09a"
Sep 08 10:15:58.423341 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:58.423302    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:58.423716 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:58.423613    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:58.436129 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:58.436107    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:15:58.865162 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:15:58.865079    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:16:04.114432 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.114401    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-338aba-29814375-95zwp"]
Sep 08 10:16:04.124842 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.124817    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-338aba-29814375-95zwp"]
Sep 08 10:16:04.153829 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.153780    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"]
Sep 08 10:16:04.154044 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.154002    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd" podUID="fc7fea57-52e5-4edc-bdef-fb5ec70e0da9" containerName="payload-processing" containerID="cri-o://eca9b48a877299d2f97201d79e6869583bdc287603de9dd5faef8556fc47f01b" gracePeriod=30
Sep 08 10:16:04.167018 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.166993    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"]
Sep 08 10:16:04.167406 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.167376    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl" podUID="33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b" containerName="payload-pre-processing" containerID="cri-o://273abd33f251dce424e96281de2f52f0508c958d38211b5c061d7729898cf981" gracePeriod=30
Sep 08 10:16:04.178726 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.178695    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl" podUID="33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.45:9004: connect: connection refused"
Sep 08 10:16:04.178835 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.178698    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd" podUID="fc7fea57-52e5-4edc-bdef-fb5ec70e0da9" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.44:9004: connect: connection refused"
Sep 08 10:16:04.369944 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.369914    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55"]
Sep 08 10:16:04.374756 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.374725    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-338aba-hsd55"]
Sep 08 10:16:04.443057 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.443032    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"
Sep 08 10:16:04.445845 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.445825    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"
Sep 08 10:16:04.563892 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.563867    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlq6d\" (UniqueName: \"kubernetes.io/projected/33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b-kube-api-access-tlq6d\") pod \"33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b\" (UID: \"33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b\") "
Sep 08 10:16:04.564044 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.563930    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7fea57-52e5-4edc-bdef-fb5ec70e0da9-plugins-config-volume\") pod \"fc7fea57-52e5-4edc-bdef-fb5ec70e0da9\" (UID: \"fc7fea57-52e5-4edc-bdef-fb5ec70e0da9\") "
Sep 08 10:16:04.564044 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.563984    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5jbm\" (UniqueName: \"kubernetes.io/projected/fc7fea57-52e5-4edc-bdef-fb5ec70e0da9-kube-api-access-d5jbm\") pod \"fc7fea57-52e5-4edc-bdef-fb5ec70e0da9\" (UID: \"fc7fea57-52e5-4edc-bdef-fb5ec70e0da9\") "
Sep 08 10:16:04.564044 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.564009    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b-plugins-config-volume\") pod \"33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b\" (UID: \"33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b\") "
Sep 08 10:16:04.564364 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.564344    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc7fea57-52e5-4edc-bdef-fb5ec70e0da9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc7fea57-52e5-4edc-bdef-fb5ec70e0da9" (UID: "fc7fea57-52e5-4edc-bdef-fb5ec70e0da9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:16:04.564456 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.564433    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b" (UID: "33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:16:04.566084 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.566064    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b-kube-api-access-tlq6d" (OuterVolumeSpecName: "kube-api-access-tlq6d") pod "33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b" (UID: "33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b"). InnerVolumeSpecName "kube-api-access-tlq6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:16:04.566143 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.566080    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7fea57-52e5-4edc-bdef-fb5ec70e0da9-kube-api-access-d5jbm" (OuterVolumeSpecName: "kube-api-access-d5jbm") pod "fc7fea57-52e5-4edc-bdef-fb5ec70e0da9" (UID: "fc7fea57-52e5-4edc-bdef-fb5ec70e0da9"). InnerVolumeSpecName "kube-api-access-d5jbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:16:04.637048 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.636987    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64c7225c-329c-42ae-a8bc-ac7e4d6cf20e" path="/var/lib/kubelet/pods/64c7225c-329c-42ae-a8bc-ac7e4d6cf20e/volumes"
Sep 08 10:16:04.637788 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.637761    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a22068f-2b6b-4cf9-bc67-6ecdd242f7aa" path="/var/lib/kubelet/pods/6a22068f-2b6b-4cf9-bc67-6ecdd242f7aa/volumes"
Sep 08 10:16:04.665400 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.665379    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5jbm\" (UniqueName: \"kubernetes.io/projected/fc7fea57-52e5-4edc-bdef-fb5ec70e0da9-kube-api-access-d5jbm\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:16:04.665400 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.665400    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:16:04.665521 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.665411    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlq6d\" (UniqueName: \"kubernetes.io/projected/33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b-kube-api-access-tlq6d\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:16:04.665521 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.665419    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7fea57-52e5-4edc-bdef-fb5ec70e0da9-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:16:04.886348 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.886319    2559 generic.go:356] "Generic (PLEG): container finished" podID="33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b" containerID="273abd33f251dce424e96281de2f52f0508c958d38211b5c061d7729898cf981" exitCode=0
Sep 08 10:16:04.886490 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.886373    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"
Sep 08 10:16:04.886490 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.886392    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl" event={"ID":"33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b","Type":"ContainerDied","Data":"273abd33f251dce424e96281de2f52f0508c958d38211b5c061d7729898cf981"}
Sep 08 10:16:04.886490 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.886432    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl" event={"ID":"33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b","Type":"ContainerDied","Data":"c822db6114156c6a71c74920242d549213e3cb12f7e5106cad76dafecb1e8a12"}
Sep 08 10:16:04.886490 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.886453    2559 scope.go:117] "RemoveContainer" containerID="273abd33f251dce424e96281de2f52f0508c958d38211b5c061d7729898cf981"
Sep 08 10:16:04.887845 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.887824    2559 generic.go:356] "Generic (PLEG): container finished" podID="fc7fea57-52e5-4edc-bdef-fb5ec70e0da9" containerID="eca9b48a877299d2f97201d79e6869583bdc287603de9dd5faef8556fc47f01b" exitCode=0
Sep 08 10:16:04.887947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.887882    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"
Sep 08 10:16:04.887947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.887915    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd" event={"ID":"fc7fea57-52e5-4edc-bdef-fb5ec70e0da9","Type":"ContainerDied","Data":"eca9b48a877299d2f97201d79e6869583bdc287603de9dd5faef8556fc47f01b"}
Sep 08 10:16:04.888051 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.887975    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd" event={"ID":"fc7fea57-52e5-4edc-bdef-fb5ec70e0da9","Type":"ContainerDied","Data":"551cc8bd31c011cc409a294d8c2372147b8f1ce2e0c307369f8847c957843bc3"}
Sep 08 10:16:04.895745 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.895727    2559 scope.go:117] "RemoveContainer" containerID="273abd33f251dce424e96281de2f52f0508c958d38211b5c061d7729898cf981"
Sep 08 10:16:04.895999 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:16:04.895979    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273abd33f251dce424e96281de2f52f0508c958d38211b5c061d7729898cf981\": container with ID starting with 273abd33f251dce424e96281de2f52f0508c958d38211b5c061d7729898cf981 not found: ID does not exist" containerID="273abd33f251dce424e96281de2f52f0508c958d38211b5c061d7729898cf981"
Sep 08 10:16:04.896063 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.896006    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273abd33f251dce424e96281de2f52f0508c958d38211b5c061d7729898cf981"} err="failed to get container status \"273abd33f251dce424e96281de2f52f0508c958d38211b5c061d7729898cf981\": rpc error: code = NotFound desc = could not find container \"273abd33f251dce424e96281de2f52f0508c958d38211b5c061d7729898cf981\": container with ID starting with 273abd33f251dce424e96281de2f52f0508c958d38211b5c061d7729898cf981 not found: ID does not exist"
Sep 08 10:16:04.896063 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.896024    2559 scope.go:117] "RemoveContainer" containerID="eca9b48a877299d2f97201d79e6869583bdc287603de9dd5faef8556fc47f01b"
Sep 08 10:16:04.904059 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.904041    2559 scope.go:117] "RemoveContainer" containerID="eca9b48a877299d2f97201d79e6869583bdc287603de9dd5faef8556fc47f01b"
Sep 08 10:16:04.904337 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:16:04.904317    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca9b48a877299d2f97201d79e6869583bdc287603de9dd5faef8556fc47f01b\": container with ID starting with eca9b48a877299d2f97201d79e6869583bdc287603de9dd5faef8556fc47f01b not found: ID does not exist" containerID="eca9b48a877299d2f97201d79e6869583bdc287603de9dd5faef8556fc47f01b"
Sep 08 10:16:04.904406 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.904346    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca9b48a877299d2f97201d79e6869583bdc287603de9dd5faef8556fc47f01b"} err="failed to get container status \"eca9b48a877299d2f97201d79e6869583bdc287603de9dd5faef8556fc47f01b\": rpc error: code = NotFound desc = could not find container \"eca9b48a877299d2f97201d79e6869583bdc287603de9dd5faef8556fc47f01b\": container with ID starting with eca9b48a877299d2f97201d79e6869583bdc287603de9dd5faef8556fc47f01b not found: ID does not exist"
Sep 08 10:16:04.924338 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.924313    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"]
Sep 08 10:16:04.929195 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.929176    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-338aba-56d4549fc4-xzqhl"]
Sep 08 10:16:04.942624 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.942602    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"]
Sep 08 10:16:04.945867 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:04.945842    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-338aba-d4c497f4f-lwvwd"]
Sep 08 10:16:06.637104 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:06.637071    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b" path="/var/lib/kubelet/pods/33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b/volumes"
Sep 08 10:16:06.637492 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:06.637480    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc7fea57-52e5-4edc-bdef-fb5ec70e0da9" path="/var/lib/kubelet/pods/fc7fea57-52e5-4edc-bdef-fb5ec70e0da9/volumes"
Sep 08 10:16:08.055918 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.055875    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd" podUID="ae079d35-f357-4b67-b143-d631b6cfe44d" containerName="main" containerID="cri-o://06514c28a317604873cfea12b739f375f2f1c519e4cd08b01340d44d14a16f48" gracePeriod=45
Sep 08 10:16:08.309287 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.309224    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:16:08.402503 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.402477    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-home\") pod \"ae079d35-f357-4b67-b143-d631b6cfe44d\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") "
Sep 08 10:16:08.402649 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.402545    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae079d35-f357-4b67-b143-d631b6cfe44d-tls-certs\") pod \"ae079d35-f357-4b67-b143-d631b6cfe44d\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") "
Sep 08 10:16:08.402649 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.402586    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-tmp-dir\") pod \"ae079d35-f357-4b67-b143-d631b6cfe44d\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") "
Sep 08 10:16:08.402649 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.402601    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-dshm\") pod \"ae079d35-f357-4b67-b143-d631b6cfe44d\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") "
Sep 08 10:16:08.402649 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.402626    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-model-cache\") pod \"ae079d35-f357-4b67-b143-d631b6cfe44d\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") "
Sep 08 10:16:08.402823 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.402668    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7mzh\" (UniqueName: \"kubernetes.io/projected/ae079d35-f357-4b67-b143-d631b6cfe44d-kube-api-access-j7mzh\") pod \"ae079d35-f357-4b67-b143-d631b6cfe44d\" (UID: \"ae079d35-f357-4b67-b143-d631b6cfe44d\") "
Sep 08 10:16:08.402823 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.402773    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-home" (OuterVolumeSpecName: "home") pod "ae079d35-f357-4b67-b143-d631b6cfe44d" (UID: "ae079d35-f357-4b67-b143-d631b6cfe44d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:16:08.402823 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.402811    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae079d35-f357-4b67-b143-d631b6cfe44d" (UID: "ae079d35-f357-4b67-b143-d631b6cfe44d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:16:08.403015 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.402992    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae079d35-f357-4b67-b143-d631b6cfe44d" (UID: "ae079d35-f357-4b67-b143-d631b6cfe44d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:16:08.403073 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.403043    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-home\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:16:08.403073 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.403058    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-tmp-dir\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:16:08.404878 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.404844    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae079d35-f357-4b67-b143-d631b6cfe44d-kube-api-access-j7mzh" (OuterVolumeSpecName: "kube-api-access-j7mzh") pod "ae079d35-f357-4b67-b143-d631b6cfe44d" (UID: "ae079d35-f357-4b67-b143-d631b6cfe44d"). InnerVolumeSpecName "kube-api-access-j7mzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:16:08.405082 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.405067    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae079d35-f357-4b67-b143-d631b6cfe44d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae079d35-f357-4b67-b143-d631b6cfe44d" (UID: "ae079d35-f357-4b67-b143-d631b6cfe44d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:16:08.405130 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.405065    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-dshm" (OuterVolumeSpecName: "dshm") pod "ae079d35-f357-4b67-b143-d631b6cfe44d" (UID: "ae079d35-f357-4b67-b143-d631b6cfe44d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:16:08.504122 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.504098    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae079d35-f357-4b67-b143-d631b6cfe44d-tls-certs\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:16:08.504122 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.504120    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-dshm\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:16:08.504288 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.504134    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae079d35-f357-4b67-b143-d631b6cfe44d-model-cache\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:16:08.504288 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.504146    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7mzh\" (UniqueName: \"kubernetes.io/projected/ae079d35-f357-4b67-b143-d631b6cfe44d-kube-api-access-j7mzh\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:16:08.913107 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.913068    2559 generic.go:356] "Generic (PLEG): container finished" podID="ae079d35-f357-4b67-b143-d631b6cfe44d" containerID="06514c28a317604873cfea12b739f375f2f1c519e4cd08b01340d44d14a16f48" exitCode=0
Sep 08 10:16:08.913346 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.913115    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"
Sep 08 10:16:08.913346 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.913166    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd" event={"ID":"ae079d35-f357-4b67-b143-d631b6cfe44d","Type":"ContainerDied","Data":"06514c28a317604873cfea12b739f375f2f1c519e4cd08b01340d44d14a16f48"}
Sep 08 10:16:08.913346 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.913215    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd" event={"ID":"ae079d35-f357-4b67-b143-d631b6cfe44d","Type":"ContainerDied","Data":"0946f525f5adbee828a65a20cac68b579de06b2335321f6314773dc0fa5637d8"}
Sep 08 10:16:08.913346 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.913237    2559 scope.go:117] "RemoveContainer" containerID="06514c28a317604873cfea12b739f375f2f1c519e4cd08b01340d44d14a16f48"
Sep 08 10:16:08.922505 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.922482    2559 scope.go:117] "RemoveContainer" containerID="06514c28a317604873cfea12b739f375f2f1c519e4cd08b01340d44d14a16f48"
Sep 08 10:16:08.922809 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:16:08.922781    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06514c28a317604873cfea12b739f375f2f1c519e4cd08b01340d44d14a16f48\": container with ID starting with 06514c28a317604873cfea12b739f375f2f1c519e4cd08b01340d44d14a16f48 not found: ID does not exist" containerID="06514c28a317604873cfea12b739f375f2f1c519e4cd08b01340d44d14a16f48"
Sep 08 10:16:08.922863 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.922814    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06514c28a317604873cfea12b739f375f2f1c519e4cd08b01340d44d14a16f48"} err="failed to get container status \"06514c28a317604873cfea12b739f375f2f1c519e4cd08b01340d44d14a16f48\": rpc error: code = NotFound desc = could not find container \"06514c28a317604873cfea12b739f375f2f1c519e4cd08b01340d44d14a16f48\": container with ID starting with 06514c28a317604873cfea12b739f375f2f1c519e4cd08b01340d44d14a16f48 not found: ID does not exist"
Sep 08 10:16:08.930891 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.930866    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"]
Sep 08 10:16:08.934654 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:08.934629    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-338aba/autores-model-338aba-kserve-6f4c95d5c9-z9zpd"]
Sep 08 10:16:10.636651 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:10.636618    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae079d35-f357-4b67-b143-d631b6cfe44d" path="/var/lib/kubelet/pods/ae079d35-f357-4b67-b143-d631b6cfe44d/volumes"
Sep 08 10:16:16.417613 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.417533    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"]
Sep 08 10:16:16.418051 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.417952    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b" containerName="payload-pre-processing"
Sep 08 10:16:16.418051 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.417964    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b" containerName="payload-pre-processing"
Sep 08 10:16:16.418051 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.417976    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae079d35-f357-4b67-b143-d631b6cfe44d" containerName="main"
Sep 08 10:16:16.418051 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.417982    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae079d35-f357-4b67-b143-d631b6cfe44d" containerName="main"
Sep 08 10:16:16.418051 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.417995    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc7fea57-52e5-4edc-bdef-fb5ec70e0da9" containerName="payload-processing"
Sep 08 10:16:16.418051 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.418001    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc7fea57-52e5-4edc-bdef-fb5ec70e0da9" containerName="payload-processing"
Sep 08 10:16:16.418051 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.418013    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c7225c-329c-42ae-a8bc-ac7e4d6cf20e" containerName="revoke-keys"
Sep 08 10:16:16.418051 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.418018    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c7225c-329c-42ae-a8bc-ac7e4d6cf20e" containerName="revoke-keys"
Sep 08 10:16:16.418333 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.418076    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc7fea57-52e5-4edc-bdef-fb5ec70e0da9" containerName="payload-processing"
Sep 08 10:16:16.418333 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.418086    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="33f34573-c6bd-4ef8-8c4d-f4b250cb9d5b" containerName="payload-pre-processing"
Sep 08 10:16:16.418333 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.418092    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae079d35-f357-4b67-b143-d631b6cfe44d" containerName="main"
Sep 08 10:16:16.418333 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.418098    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="64c7225c-329c-42ae-a8bc-ac7e4d6cf20e" containerName="revoke-keys"
Sep 08 10:16:16.422670 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.422653    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"
Sep 08 10:16:16.426676 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.426636    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-9c734c\""
Sep 08 10:16:16.427029 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.427012    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-9c734c-dockercfg-pldct\""
Sep 08 10:16:16.439374 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.439351    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"]
Sep 08 10:16:16.446885 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.446860    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"]
Sep 08 10:16:16.451136 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.451117    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"
Sep 08 10:16:16.471259 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.471213    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"]
Sep 08 10:16:16.576705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.576672    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trh6d\" (UniqueName: \"kubernetes.io/projected/9cf556e6-58c9-428d-a9f3-883b9dc89de4-kube-api-access-trh6d\") pod \"payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb\" (UID: \"9cf556e6-58c9-428d-a9f3-883b9dc89de4\") " pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"
Sep 08 10:16:16.576859 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.576715    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c9f58d-a799-4234-af58-f6bcfc8496a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27\" (UID: \"f8c9f58d-a799-4234-af58-f6bcfc8496a1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"
Sep 08 10:16:16.576859 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.576738    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf556e6-58c9-428d-a9f3-883b9dc89de4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb\" (UID: \"9cf556e6-58c9-428d-a9f3-883b9dc89de4\") " pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"
Sep 08 10:16:16.576859 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.576832    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnnws\" (UniqueName: \"kubernetes.io/projected/f8c9f58d-a799-4234-af58-f6bcfc8496a1-kube-api-access-cnnws\") pod \"payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27\" (UID: \"f8c9f58d-a799-4234-af58-f6bcfc8496a1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"
Sep 08 10:16:16.678025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.677945    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trh6d\" (UniqueName: \"kubernetes.io/projected/9cf556e6-58c9-428d-a9f3-883b9dc89de4-kube-api-access-trh6d\") pod \"payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb\" (UID: \"9cf556e6-58c9-428d-a9f3-883b9dc89de4\") " pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"
Sep 08 10:16:16.678025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.677989    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c9f58d-a799-4234-af58-f6bcfc8496a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27\" (UID: \"f8c9f58d-a799-4234-af58-f6bcfc8496a1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"
Sep 08 10:16:16.678025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.678008    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf556e6-58c9-428d-a9f3-883b9dc89de4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb\" (UID: \"9cf556e6-58c9-428d-a9f3-883b9dc89de4\") " pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"
Sep 08 10:16:16.678279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.678044    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnws\" (UniqueName: \"kubernetes.io/projected/f8c9f58d-a799-4234-af58-f6bcfc8496a1-kube-api-access-cnnws\") pod \"payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27\" (UID: \"f8c9f58d-a799-4234-af58-f6bcfc8496a1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"
Sep 08 10:16:16.678784 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.678765    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c9f58d-a799-4234-af58-f6bcfc8496a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27\" (UID: \"f8c9f58d-a799-4234-af58-f6bcfc8496a1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"
Sep 08 10:16:16.678825 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.678764    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf556e6-58c9-428d-a9f3-883b9dc89de4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb\" (UID: \"9cf556e6-58c9-428d-a9f3-883b9dc89de4\") " pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"
Sep 08 10:16:16.687778 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.687750    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnws\" (UniqueName: \"kubernetes.io/projected/f8c9f58d-a799-4234-af58-f6bcfc8496a1-kube-api-access-cnnws\") pod \"payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27\" (UID: \"f8c9f58d-a799-4234-af58-f6bcfc8496a1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"
Sep 08 10:16:16.687961 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.687939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trh6d\" (UniqueName: \"kubernetes.io/projected/9cf556e6-58c9-428d-a9f3-883b9dc89de4-kube-api-access-trh6d\") pod \"payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb\" (UID: \"9cf556e6-58c9-428d-a9f3-883b9dc89de4\") " pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"
Sep 08 10:16:16.734662 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.734639    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"
Sep 08 10:16:16.762857 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:16.762827    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"
Sep 08 10:16:17.024546 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:16:17.024523    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c9f58d_a799_4234_af58_f6bcfc8496a1.slice/crio-bca19a50af6616029b940952a038614679ac1b272588dc10e6e67e4971bc6e74 WatchSource:0}: Error finding container bca19a50af6616029b940952a038614679ac1b272588dc10e6e67e4971bc6e74: Status 404 returned error can't find the container with id bca19a50af6616029b940952a038614679ac1b272588dc10e6e67e4971bc6e74
Sep 08 10:16:17.025750 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:17.025728    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"]
Sep 08 10:16:17.099080 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:17.099049    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"]
Sep 08 10:16:17.106004 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:16:17.105968    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf556e6_58c9_428d_a9f3_883b9dc89de4.slice/crio-6efb9b310ce0ad27f0fb7c63d92fdec45db1938eac9bc30fd2c3b380502d9db1 WatchSource:0}: Error finding container 6efb9b310ce0ad27f0fb7c63d92fdec45db1938eac9bc30fd2c3b380502d9db1: Status 404 returned error can't find the container with id 6efb9b310ce0ad27f0fb7c63d92fdec45db1938eac9bc30fd2c3b380502d9db1
Sep 08 10:16:17.966183 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:17.966149    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb" event={"ID":"9cf556e6-58c9-428d-a9f3-883b9dc89de4","Type":"ContainerStarted","Data":"020ced3789330cdb13d21b5184c6138a0bed4af646885e9883c5a33e1dcfd232"}
Sep 08 10:16:17.966653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:17.966213    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb" event={"ID":"9cf556e6-58c9-428d-a9f3-883b9dc89de4","Type":"ContainerStarted","Data":"6efb9b310ce0ad27f0fb7c63d92fdec45db1938eac9bc30fd2c3b380502d9db1"}
Sep 08 10:16:17.966653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:17.966282    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"
Sep 08 10:16:17.967922 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:17.967897    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27" event={"ID":"f8c9f58d-a799-4234-af58-f6bcfc8496a1","Type":"ContainerStarted","Data":"835338eb86e4902e604bc73f69e8cd05c089ab69e682a5dd4cf882c3426b1198"}
Sep 08 10:16:17.968047 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:17.967936    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27" event={"ID":"f8c9f58d-a799-4234-af58-f6bcfc8496a1","Type":"ContainerStarted","Data":"bca19a50af6616029b940952a038614679ac1b272588dc10e6e67e4971bc6e74"}
Sep 08 10:16:17.968047 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:17.968019    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"
Sep 08 10:16:17.992489 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:17.992444    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb" podStartSLOduration=1.992431801 podStartE2EDuration="1.992431801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:16:17.991907479 +0000 UTC m=+897.833131887" watchObservedRunningTime="2026-09-08 10:16:17.992431801 +0000 UTC m=+897.833656231"
Sep 08 10:16:18.012977 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:18.012934    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27" podStartSLOduration=2.012918709 podStartE2EDuration="2.012918709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:16:18.01244753 +0000 UTC m=+897.853671932" watchObservedRunningTime="2026-09-08 10:16:18.012918709 +0000 UTC m=+897.854143095"
Sep 08 10:16:20.655982 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:20.655954    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:16:20.656427 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:20.656314    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:16:22.755823 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:22.755782    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4"]
Sep 08 10:16:22.760969 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:22.760951    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4"
Sep 08 10:16:22.765285 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:22.765239    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:16:22.765402 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:22.765314    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:16:22.777379 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:22.777346    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4"]
Sep 08 10:16:22.831842 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:22.831815    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a652c0c9-82c6-45b4-9001-73fa6ac2adb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-057291ac-vvpz4\" (UID: \"a652c0c9-82c6-45b4-9001-73fa6ac2adb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4"
Sep 08 10:16:22.932455 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:22.932429    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a652c0c9-82c6-45b4-9001-73fa6ac2adb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-057291ac-vvpz4\" (UID: \"a652c0c9-82c6-45b4-9001-73fa6ac2adb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4"
Sep 08 10:16:22.932994 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:22.932975    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a652c0c9-82c6-45b4-9001-73fa6ac2adb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-057291ac-vvpz4\" (UID: \"a652c0c9-82c6-45b4-9001-73fa6ac2adb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4"
Sep 08 10:16:23.070886 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:23.070827    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4"
Sep 08 10:16:23.340942 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:23.340917    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4"]
Sep 08 10:16:23.342233 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:16:23.342206    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda652c0c9_82c6_45b4_9001_73fa6ac2adb3.slice/crio-563fd0c117217413a1161b2de5de058ad7403fef511938e667d521426cbed588 WatchSource:0}: Error finding container 563fd0c117217413a1161b2de5de058ad7403fef511938e667d521426cbed588: Status 404 returned error can't find the container with id 563fd0c117217413a1161b2de5de058ad7403fef511938e667d521426cbed588
Sep 08 10:16:24.003026 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:24.002982    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4" event={"ID":"a652c0c9-82c6-45b4-9001-73fa6ac2adb3","Type":"ContainerStarted","Data":"297bc6390e3d5adcb064f7cb2401e94fe2dd44beb42f4f9c8a156c52355dd68c"}
Sep 08 10:16:24.003423 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:24.003050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4" event={"ID":"a652c0c9-82c6-45b4-9001-73fa6ac2adb3","Type":"ContainerStarted","Data":"563fd0c117217413a1161b2de5de058ad7403fef511938e667d521426cbed588"}
Sep 08 10:16:24.020122 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:24.020071    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4" podStartSLOduration=2.020053467 podStartE2EDuration="2.020053467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:16:24.019721038 +0000 UTC m=+903.860945441" watchObservedRunningTime="2026-09-08 10:16:24.020053467 +0000 UTC m=+903.861277851"
Sep 08 10:16:25.010382 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:25.010347    2559 generic.go:356] "Generic (PLEG): container finished" podID="a652c0c9-82c6-45b4-9001-73fa6ac2adb3" containerID="297bc6390e3d5adcb064f7cb2401e94fe2dd44beb42f4f9c8a156c52355dd68c" exitCode=0
Sep 08 10:16:25.010791 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:25.010395    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4" event={"ID":"a652c0c9-82c6-45b4-9001-73fa6ac2adb3","Type":"ContainerDied","Data":"297bc6390e3d5adcb064f7cb2401e94fe2dd44beb42f4f9c8a156c52355dd68c"}
Sep 08 10:16:26.152898 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:26.152872    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4"
Sep 08 10:16:26.262339 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:26.262310    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a652c0c9-82c6-45b4-9001-73fa6ac2adb3-maas-api-service-ca\") pod \"a652c0c9-82c6-45b4-9001-73fa6ac2adb3\" (UID: \"a652c0c9-82c6-45b4-9001-73fa6ac2adb3\") "
Sep 08 10:16:26.262640 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:26.262619    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a652c0c9-82c6-45b4-9001-73fa6ac2adb3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a652c0c9-82c6-45b4-9001-73fa6ac2adb3" (UID: "a652c0c9-82c6-45b4-9001-73fa6ac2adb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:16:26.363827 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:26.363756    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a652c0c9-82c6-45b4-9001-73fa6ac2adb3-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:16:27.022540 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:27.022512    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4"
Sep 08 10:16:27.022730 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:27.022557    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4" event={"ID":"a652c0c9-82c6-45b4-9001-73fa6ac2adb3","Type":"ContainerDied","Data":"563fd0c117217413a1161b2de5de058ad7403fef511938e667d521426cbed588"}
Sep 08 10:16:27.022730 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:27.022616    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="563fd0c117217413a1161b2de5de058ad7403fef511938e667d521426cbed588"
Sep 08 10:16:28.975974 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:28.975944    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"
Sep 08 10:16:28.976390 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:28.976013    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"
Sep 08 10:16:29.847888 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:29.847856    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4"]
Sep 08 10:16:29.851188 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:29.851157    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-057291ac-vvpz4"]
Sep 08 10:16:30.637052 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:16:30.637020    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a652c0c9-82c6-45b4-9001-73fa6ac2adb3" path="/var/lib/kubelet/pods/a652c0c9-82c6-45b4-9001-73fa6ac2adb3/volumes"
Sep 08 10:17:35.569560 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:35.569463    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds"]
Sep 08 10:17:35.570365 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:35.570340    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a652c0c9-82c6-45b4-9001-73fa6ac2adb3" containerName="revoke-keys"
Sep 08 10:17:35.570365 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:35.570365    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a652c0c9-82c6-45b4-9001-73fa6ac2adb3" containerName="revoke-keys"
Sep 08 10:17:35.570516 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:35.570446    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a652c0c9-82c6-45b4-9001-73fa6ac2adb3" containerName="revoke-keys"
Sep 08 10:17:35.573948 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:35.573924    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds"
Sep 08 10:17:35.574754 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:35.574732    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds"]
Sep 08 10:17:35.577322 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:35.577289    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:17:35.577322 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:35.577305    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:17:35.671255 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:35.671217    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9c734c-hrdds\" (UID: \"d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds"
Sep 08 10:17:35.772367 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:35.772329    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9c734c-hrdds\" (UID: \"d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds"
Sep 08 10:17:35.772917 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:35.772897    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-9c734c-hrdds\" (UID: \"d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds"
Sep 08 10:17:35.885578 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:35.885517    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds"
Sep 08 10:17:36.169458 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:36.169433    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds"]
Sep 08 10:17:36.171393 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:17:36.171363    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd123fc4c_e49b_4c1a_9ed0_d5727d1fbc6e.slice/crio-4543c8b3be7b3c594186d5d34ca23c84fd5e0817417c286b37b39f8cf2394f1e WatchSource:0}: Error finding container 4543c8b3be7b3c594186d5d34ca23c84fd5e0817417c286b37b39f8cf2394f1e: Status 404 returned error can't find the container with id 4543c8b3be7b3c594186d5d34ca23c84fd5e0817417c286b37b39f8cf2394f1e
Sep 08 10:17:36.409933 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:36.409895    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds" event={"ID":"d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e","Type":"ContainerStarted","Data":"d0f657bb6a047c0747c9eb6ecdcf9536377b6820f629c3015281ef1598628b5e"}
Sep 08 10:17:36.410055 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:36.409954    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds" event={"ID":"d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e","Type":"ContainerStarted","Data":"4543c8b3be7b3c594186d5d34ca23c84fd5e0817417c286b37b39f8cf2394f1e"}
Sep 08 10:17:36.429680 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:36.429602    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds" podStartSLOduration=1.4295874259999999 podStartE2EDuration="1.429587426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:17:36.428277596 +0000 UTC m=+976.269501997" watchObservedRunningTime="2026-09-08 10:17:36.429587426 +0000 UTC m=+976.270811810"
Sep 08 10:17:37.416584 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:37.416547    2559 generic.go:356] "Generic (PLEG): container finished" podID="d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e" containerID="d0f657bb6a047c0747c9eb6ecdcf9536377b6820f629c3015281ef1598628b5e" exitCode=0
Sep 08 10:17:37.416943 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:37.416605    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds" event={"ID":"d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e","Type":"ContainerDied","Data":"d0f657bb6a047c0747c9eb6ecdcf9536377b6820f629c3015281ef1598628b5e"}
Sep 08 10:17:38.556299 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:38.556273    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds"
Sep 08 10:17:38.701535 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:38.701467    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e-maas-api-service-ca\") pod \"d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e\" (UID: \"d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e\") "
Sep 08 10:17:38.701793 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:38.701772    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e" (UID: "d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:17:38.802626 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:38.802597    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:17:39.429098 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:39.429067    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds"
Sep 08 10:17:39.429098 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:39.429074    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds" event={"ID":"d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e","Type":"ContainerDied","Data":"4543c8b3be7b3c594186d5d34ca23c84fd5e0817417c286b37b39f8cf2394f1e"}
Sep 08 10:17:39.429346 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:39.429126    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4543c8b3be7b3c594186d5d34ca23c84fd5e0817417c286b37b39f8cf2394f1e"
Sep 08 10:17:46.318981 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.318900    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"]
Sep 08 10:17:46.319413 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.319144    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb" podUID="9cf556e6-58c9-428d-a9f3-883b9dc89de4" containerName="payload-processing" containerID="cri-o://020ced3789330cdb13d21b5184c6138a0bed4af646885e9883c5a33e1dcfd232" gracePeriod=30
Sep 08 10:17:46.328538 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.328509    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"]
Sep 08 10:17:46.328756 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.328722    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27" podUID="f8c9f58d-a799-4234-af58-f6bcfc8496a1" containerName="payload-pre-processing" containerID="cri-o://835338eb86e4902e604bc73f69e8cd05c089ab69e682a5dd4cf882c3426b1198" gracePeriod=30
Sep 08 10:17:46.478544 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.478507    2559 generic.go:356] "Generic (PLEG): container finished" podID="9cf556e6-58c9-428d-a9f3-883b9dc89de4" containerID="020ced3789330cdb13d21b5184c6138a0bed4af646885e9883c5a33e1dcfd232" exitCode=0
Sep 08 10:17:46.478718 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.478596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb" event={"ID":"9cf556e6-58c9-428d-a9f3-883b9dc89de4","Type":"ContainerDied","Data":"020ced3789330cdb13d21b5184c6138a0bed4af646885e9883c5a33e1dcfd232"}
Sep 08 10:17:46.480364 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.480287    2559 generic.go:356] "Generic (PLEG): container finished" podID="f8c9f58d-a799-4234-af58-f6bcfc8496a1" containerID="835338eb86e4902e604bc73f69e8cd05c089ab69e682a5dd4cf882c3426b1198" exitCode=0
Sep 08 10:17:46.480364 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.480354    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27" event={"ID":"f8c9f58d-a799-4234-af58-f6bcfc8496a1","Type":"ContainerDied","Data":"835338eb86e4902e604bc73f69e8cd05c089ab69e682a5dd4cf882c3426b1198"}
Sep 08 10:17:46.507057 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.507027    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds"]
Sep 08 10:17:46.512027 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.512004    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-9c734c-hrdds"]
Sep 08 10:17:46.603921 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.603896    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"
Sep 08 10:17:46.606856 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.606838    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"
Sep 08 10:17:46.637160 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.637132    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e" path="/var/lib/kubelet/pods/d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e/volumes"
Sep 08 10:17:46.774779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.774748    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnnws\" (UniqueName: \"kubernetes.io/projected/f8c9f58d-a799-4234-af58-f6bcfc8496a1-kube-api-access-cnnws\") pod \"f8c9f58d-a799-4234-af58-f6bcfc8496a1\" (UID: \"f8c9f58d-a799-4234-af58-f6bcfc8496a1\") "
Sep 08 10:17:46.774951 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.774837    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf556e6-58c9-428d-a9f3-883b9dc89de4-plugins-config-volume\") pod \"9cf556e6-58c9-428d-a9f3-883b9dc89de4\" (UID: \"9cf556e6-58c9-428d-a9f3-883b9dc89de4\") "
Sep 08 10:17:46.774951 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.774874    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trh6d\" (UniqueName: \"kubernetes.io/projected/9cf556e6-58c9-428d-a9f3-883b9dc89de4-kube-api-access-trh6d\") pod \"9cf556e6-58c9-428d-a9f3-883b9dc89de4\" (UID: \"9cf556e6-58c9-428d-a9f3-883b9dc89de4\") "
Sep 08 10:17:46.774951 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.774935    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c9f58d-a799-4234-af58-f6bcfc8496a1-plugins-config-volume\") pod \"f8c9f58d-a799-4234-af58-f6bcfc8496a1\" (UID: \"f8c9f58d-a799-4234-af58-f6bcfc8496a1\") "
Sep 08 10:17:46.775328 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.775299    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf556e6-58c9-428d-a9f3-883b9dc89de4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9cf556e6-58c9-428d-a9f3-883b9dc89de4" (UID: "9cf556e6-58c9-428d-a9f3-883b9dc89de4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:17:46.775328 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.775311    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c9f58d-a799-4234-af58-f6bcfc8496a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8c9f58d-a799-4234-af58-f6bcfc8496a1" (UID: "f8c9f58d-a799-4234-af58-f6bcfc8496a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:17:46.777030 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.777005    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf556e6-58c9-428d-a9f3-883b9dc89de4-kube-api-access-trh6d" (OuterVolumeSpecName: "kube-api-access-trh6d") pod "9cf556e6-58c9-428d-a9f3-883b9dc89de4" (UID: "9cf556e6-58c9-428d-a9f3-883b9dc89de4"). InnerVolumeSpecName "kube-api-access-trh6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:17:46.777107 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.777032    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c9f58d-a799-4234-af58-f6bcfc8496a1-kube-api-access-cnnws" (OuterVolumeSpecName: "kube-api-access-cnnws") pod "f8c9f58d-a799-4234-af58-f6bcfc8496a1" (UID: "f8c9f58d-a799-4234-af58-f6bcfc8496a1"). InnerVolumeSpecName "kube-api-access-cnnws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:17:46.876663 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.876601    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c9f58d-a799-4234-af58-f6bcfc8496a1-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:17:46.876663 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.876624    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnnws\" (UniqueName: \"kubernetes.io/projected/f8c9f58d-a799-4234-af58-f6bcfc8496a1-kube-api-access-cnnws\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:17:46.876663 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.876634    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf556e6-58c9-428d-a9f3-883b9dc89de4-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:17:46.876663 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:46.876643    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trh6d\" (UniqueName: \"kubernetes.io/projected/9cf556e6-58c9-428d-a9f3-883b9dc89de4-kube-api-access-trh6d\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:17:47.495076 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:47.495040    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb" event={"ID":"9cf556e6-58c9-428d-a9f3-883b9dc89de4","Type":"ContainerDied","Data":"6efb9b310ce0ad27f0fb7c63d92fdec45db1938eac9bc30fd2c3b380502d9db1"}
Sep 08 10:17:47.495076 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:47.495063    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"
Sep 08 10:17:47.495639 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:47.495143    2559 scope.go:117] "RemoveContainer" containerID="020ced3789330cdb13d21b5184c6138a0bed4af646885e9883c5a33e1dcfd232"
Sep 08 10:17:47.496715 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:47.496695    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"
Sep 08 10:17:47.496818 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:47.496717    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27" event={"ID":"f8c9f58d-a799-4234-af58-f6bcfc8496a1","Type":"ContainerDied","Data":"bca19a50af6616029b940952a038614679ac1b272588dc10e6e67e4971bc6e74"}
Sep 08 10:17:47.506799 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:47.506781    2559 scope.go:117] "RemoveContainer" containerID="835338eb86e4902e604bc73f69e8cd05c089ab69e682a5dd4cf882c3426b1198"
Sep 08 10:17:47.527027 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:47.527003    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"]
Sep 08 10:17:47.531955 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:47.531932    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9c734c-cc94fd4fd-28g27"]
Sep 08 10:17:47.552526 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:47.552481    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"]
Sep 08 10:17:47.554821 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:47.554801    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9c734c-bcbf5b6fc-jqrbb"]
Sep 08 10:17:48.637540 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:48.637511    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cf556e6-58c9-428d-a9f3-883b9dc89de4" path="/var/lib/kubelet/pods/9cf556e6-58c9-428d-a9f3-883b9dc89de4/volumes"
Sep 08 10:17:48.637908 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:17:48.637853    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8c9f58d-a799-4234-af58-f6bcfc8496a1" path="/var/lib/kubelet/pods/f8c9f58d-a799-4234-af58-f6bcfc8496a1/volumes"
Sep 08 10:18:21.249406 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.249375    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"]
Sep 08 10:18:21.249803 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.249790    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c9f58d-a799-4234-af58-f6bcfc8496a1" containerName="payload-pre-processing"
Sep 08 10:18:21.249803 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.249802    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c9f58d-a799-4234-af58-f6bcfc8496a1" containerName="payload-pre-processing"
Sep 08 10:18:21.249876 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.249822    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cf556e6-58c9-428d-a9f3-883b9dc89de4" containerName="payload-processing"
Sep 08 10:18:21.249876 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.249829    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cf556e6-58c9-428d-a9f3-883b9dc89de4" containerName="payload-processing"
Sep 08 10:18:21.249876 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.249836    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e" containerName="revoke-keys"
Sep 08 10:18:21.249876 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.249841    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e" containerName="revoke-keys"
Sep 08 10:18:21.250000 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.249904    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c9f58d-a799-4234-af58-f6bcfc8496a1" containerName="payload-pre-processing"
Sep 08 10:18:21.250000 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.249910    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d123fc4c-e49b-4c1a-9ed0-d5727d1fbc6e" containerName="revoke-keys"
Sep 08 10:18:21.250000 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.249918    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cf556e6-58c9-428d-a9f3-883b9dc89de4" containerName="payload-processing"
Sep 08 10:18:21.254189 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.254172    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.257960 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.257936    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-738ee5\"/\"openshift-service-ca.crt\""
Sep 08 10:18:21.258902 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.258888    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-738ee5\"/\"default-dockercfg-fwsrk\""
Sep 08 10:18:21.258978 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.258919    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-738ee5\"/\"kube-root-ca.crt\""
Sep 08 10:18:21.259027 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.258986    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-738ee5\"/\"noten-model-738ee5-kserve-self-signed-certs\""
Sep 08 10:18:21.270580 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.270550    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"]
Sep 08 10:18:21.381988 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.381962    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.382151 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.381999    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-home\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.382151 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.382023    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-dshm\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.382151 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.382041    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-tmp-dir\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.382342 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.382154    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbzqb\" (UniqueName: \"kubernetes.io/projected/4cf02cae-a839-4125-aab2-a05efcd97c6b-kube-api-access-kbzqb\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.382342 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.382214    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-model-cache\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.483015 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.482989    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.483146 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.483027    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-home\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.483146 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.483048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-dshm\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.483146 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.483063    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-tmp-dir\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.483146 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.483117    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbzqb\" (UniqueName: \"kubernetes.io/projected/4cf02cae-a839-4125-aab2-a05efcd97c6b-kube-api-access-kbzqb\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.483410 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.483163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-model-cache\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.483496 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.483474    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-home\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.483574 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.483499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-tmp-dir\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.483621 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.483575    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-model-cache\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.485285 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.485267    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-dshm\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.485556 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.485540    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.494115 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.494092    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbzqb\" (UniqueName: \"kubernetes.io/projected/4cf02cae-a839-4125-aab2-a05efcd97c6b-kube-api-access-kbzqb\") pod \"noten-model-738ee5-kserve-7b47788df9-jndf5\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") " pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.562874 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.562786    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:21.839730 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:21.839519    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"]
Sep 08 10:18:21.842446 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:18:21.842419    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cf02cae_a839_4125_aab2_a05efcd97c6b.slice/crio-bebe2e572813cfac9c5fb74f636858c309db4ca10032073e529c6a11231794cc WatchSource:0}: Error finding container bebe2e572813cfac9c5fb74f636858c309db4ca10032073e529c6a11231794cc: Status 404 returned error can't find the container with id bebe2e572813cfac9c5fb74f636858c309db4ca10032073e529c6a11231794cc
Sep 08 10:18:22.692007 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:22.691976    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5" event={"ID":"4cf02cae-a839-4125-aab2-a05efcd97c6b","Type":"ContainerStarted","Data":"998796e74d3e45dd3f3b4d08666bc69448a694a85ca9a9e6838581c6bd310713"}
Sep 08 10:18:22.692388 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:22.692018    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5" event={"ID":"4cf02cae-a839-4125-aab2-a05efcd97c6b","Type":"ContainerStarted","Data":"bebe2e572813cfac9c5fb74f636858c309db4ca10032073e529c6a11231794cc"}
Sep 08 10:18:22.718469 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:22.718423    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5" podStartSLOduration=1.718408274 podStartE2EDuration="1.718408274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:18:22.718044749 +0000 UTC m=+1022.559269157" watchObservedRunningTime="2026-09-08 10:18:22.718408274 +0000 UTC m=+1022.559632657"
Sep 08 10:18:31.398345 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:31.398311    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"]
Sep 08 10:18:31.563415 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:31.563377    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:31.563605 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:31.563434    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:31.576024 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:31.575999    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:31.752629 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:31.752604    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:32.363683 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:32.363655    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"]
Sep 08 10:18:32.890283 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:32.890233    2559 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-self-signed-certs: secret "noten-model-738ee5-kserve-self-signed-certs" not found
Sep 08 10:18:32.890724 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:32.890341    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs podName:4cf02cae-a839-4125-aab2-a05efcd97c6b nodeName:}" failed. No retries permitted until 2026-09-08 10:18:33.390320259 +0000 UTC m=+1033.231544620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs") pod "noten-model-738ee5-kserve-7b47788df9-jndf5" (UID: "4cf02cae-a839-4125-aab2-a05efcd97c6b") : secret "noten-model-738ee5-kserve-self-signed-certs" not found
Sep 08 10:18:33.395126 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:33.395094    2559 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-self-signed-certs: secret "noten-model-738ee5-kserve-self-signed-certs" not found
Sep 08 10:18:33.395321 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:33.395171    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs podName:4cf02cae-a839-4125-aab2-a05efcd97c6b nodeName:}" failed. No retries permitted until 2026-09-08 10:18:34.395156081 +0000 UTC m=+1034.236380446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs") pod "noten-model-738ee5-kserve-7b47788df9-jndf5" (UID: "4cf02cae-a839-4125-aab2-a05efcd97c6b") : secret "noten-model-738ee5-kserve-self-signed-certs" not found
Sep 08 10:18:34.403337 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:34.403297    2559 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-self-signed-certs: secret "noten-model-738ee5-kserve-self-signed-certs" not found
Sep 08 10:18:34.403793 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:34.403389    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs podName:4cf02cae-a839-4125-aab2-a05efcd97c6b nodeName:}" failed. No retries permitted until 2026-09-08 10:18:36.403369685 +0000 UTC m=+1036.244594057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs") pod "noten-model-738ee5-kserve-7b47788df9-jndf5" (UID: "4cf02cae-a839-4125-aab2-a05efcd97c6b") : secret "noten-model-738ee5-kserve-self-signed-certs" not found
Sep 08 10:18:36.420154 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:36.420121    2559 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-self-signed-certs: secret "noten-model-738ee5-kserve-self-signed-certs" not found
Sep 08 10:18:36.420551 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:36.420190    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs podName:4cf02cae-a839-4125-aab2-a05efcd97c6b nodeName:}" failed. No retries permitted until 2026-09-08 10:18:40.420174797 +0000 UTC m=+1040.261399169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs") pod "noten-model-738ee5-kserve-7b47788df9-jndf5" (UID: "4cf02cae-a839-4125-aab2-a05efcd97c6b") : secret "noten-model-738ee5-kserve-self-signed-certs" not found
Sep 08 10:18:40.459912 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:40.459881    2559 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-self-signed-certs: secret "noten-model-738ee5-kserve-self-signed-certs" not found
Sep 08 10:18:40.460277 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:40.459946    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs podName:4cf02cae-a839-4125-aab2-a05efcd97c6b nodeName:}" failed. No retries permitted until 2026-09-08 10:18:48.459929913 +0000 UTC m=+1048.301154275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs") pod "noten-model-738ee5-kserve-7b47788df9-jndf5" (UID: "4cf02cae-a839-4125-aab2-a05efcd97c6b") : secret "noten-model-738ee5-kserve-self-signed-certs" not found
Sep 08 10:18:44.809789 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:44.809760    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"]
Sep 08 10:18:44.810218 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:44.809970    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q" podUID="0585fb9c-2d5e-48f8-a142-19cf206cbfa6" containerName="payload-processing" containerID="cri-o://a6de3ca1d82ad2da102e4252a22846765b71a69e8adc4d797bcc0e1119ddcb98" gracePeriod=30
Sep 08 10:18:44.831947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:44.829567    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"]
Sep 08 10:18:44.831947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:44.829821    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9" podUID="3d3559dc-7cce-42cd-9fd0-86afd305cd2e" containerName="payload-pre-processing" containerID="cri-o://39b49ce63d5774293bfcb85f5506b009fd2bbad90874718eac9bd3aca494586a" gracePeriod=30
Sep 08 10:18:45.108116 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.108093    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"
Sep 08 10:18:45.111088 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.111070    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"
Sep 08 10:18:45.201144 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.201118    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nqnm\" (UniqueName: \"kubernetes.io/projected/0585fb9c-2d5e-48f8-a142-19cf206cbfa6-kube-api-access-7nqnm\") pod \"0585fb9c-2d5e-48f8-a142-19cf206cbfa6\" (UID: \"0585fb9c-2d5e-48f8-a142-19cf206cbfa6\") "
Sep 08 10:18:45.201321 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.201156    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl799\" (UniqueName: \"kubernetes.io/projected/3d3559dc-7cce-42cd-9fd0-86afd305cd2e-kube-api-access-wl799\") pod \"3d3559dc-7cce-42cd-9fd0-86afd305cd2e\" (UID: \"3d3559dc-7cce-42cd-9fd0-86afd305cd2e\") "
Sep 08 10:18:45.201321 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.201189    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0585fb9c-2d5e-48f8-a142-19cf206cbfa6-plugins-config-volume\") pod \"0585fb9c-2d5e-48f8-a142-19cf206cbfa6\" (UID: \"0585fb9c-2d5e-48f8-a142-19cf206cbfa6\") "
Sep 08 10:18:45.201321 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.201212    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3559dc-7cce-42cd-9fd0-86afd305cd2e-plugins-config-volume\") pod \"3d3559dc-7cce-42cd-9fd0-86afd305cd2e\" (UID: \"3d3559dc-7cce-42cd-9fd0-86afd305cd2e\") "
Sep 08 10:18:45.201686 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.201649    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3559dc-7cce-42cd-9fd0-86afd305cd2e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d3559dc-7cce-42cd-9fd0-86afd305cd2e" (UID: "3d3559dc-7cce-42cd-9fd0-86afd305cd2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:18:45.201686 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.201654    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0585fb9c-2d5e-48f8-a142-19cf206cbfa6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0585fb9c-2d5e-48f8-a142-19cf206cbfa6" (UID: "0585fb9c-2d5e-48f8-a142-19cf206cbfa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:18:45.203174 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.203148    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3559dc-7cce-42cd-9fd0-86afd305cd2e-kube-api-access-wl799" (OuterVolumeSpecName: "kube-api-access-wl799") pod "3d3559dc-7cce-42cd-9fd0-86afd305cd2e" (UID: "3d3559dc-7cce-42cd-9fd0-86afd305cd2e"). InnerVolumeSpecName "kube-api-access-wl799". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:18:45.203302 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.203268    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0585fb9c-2d5e-48f8-a142-19cf206cbfa6-kube-api-access-7nqnm" (OuterVolumeSpecName: "kube-api-access-7nqnm") pod "0585fb9c-2d5e-48f8-a142-19cf206cbfa6" (UID: "0585fb9c-2d5e-48f8-a142-19cf206cbfa6"). InnerVolumeSpecName "kube-api-access-7nqnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:18:45.302856 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.302828    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nqnm\" (UniqueName: \"kubernetes.io/projected/0585fb9c-2d5e-48f8-a142-19cf206cbfa6-kube-api-access-7nqnm\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:45.302856 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.302852    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl799\" (UniqueName: \"kubernetes.io/projected/3d3559dc-7cce-42cd-9fd0-86afd305cd2e-kube-api-access-wl799\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:45.302856 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.302861    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0585fb9c-2d5e-48f8-a142-19cf206cbfa6-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:45.303075 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.302870    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3559dc-7cce-42cd-9fd0-86afd305cd2e-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:45.821300 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.821265    2559 generic.go:356] "Generic (PLEG): container finished" podID="3d3559dc-7cce-42cd-9fd0-86afd305cd2e" containerID="39b49ce63d5774293bfcb85f5506b009fd2bbad90874718eac9bd3aca494586a" exitCode=0
Sep 08 10:18:45.821300 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.821296    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"
Sep 08 10:18:45.821782 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.821343    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9" event={"ID":"3d3559dc-7cce-42cd-9fd0-86afd305cd2e","Type":"ContainerDied","Data":"39b49ce63d5774293bfcb85f5506b009fd2bbad90874718eac9bd3aca494586a"}
Sep 08 10:18:45.821782 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.821406    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9" event={"ID":"3d3559dc-7cce-42cd-9fd0-86afd305cd2e","Type":"ContainerDied","Data":"be36a723f1a4e32b3b58469e5bc35af91ddf55c1fb3bf524e77157f29f59b53d"}
Sep 08 10:18:45.821782 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.821440    2559 scope.go:117] "RemoveContainer" containerID="39b49ce63d5774293bfcb85f5506b009fd2bbad90874718eac9bd3aca494586a"
Sep 08 10:18:45.822798 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.822674    2559 generic.go:356] "Generic (PLEG): container finished" podID="0585fb9c-2d5e-48f8-a142-19cf206cbfa6" containerID="a6de3ca1d82ad2da102e4252a22846765b71a69e8adc4d797bcc0e1119ddcb98" exitCode=0
Sep 08 10:18:45.822798 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.822738    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"
Sep 08 10:18:45.822798 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.822760    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q" event={"ID":"0585fb9c-2d5e-48f8-a142-19cf206cbfa6","Type":"ContainerDied","Data":"a6de3ca1d82ad2da102e4252a22846765b71a69e8adc4d797bcc0e1119ddcb98"}
Sep 08 10:18:45.822967 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.822813    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q" event={"ID":"0585fb9c-2d5e-48f8-a142-19cf206cbfa6","Type":"ContainerDied","Data":"eaa6aebf397f76ea1aa10932756233b2e1700da1ccd3827bd3af657e6d28b3b7"}
Sep 08 10:18:45.837014 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.836997    2559 scope.go:117] "RemoveContainer" containerID="39b49ce63d5774293bfcb85f5506b009fd2bbad90874718eac9bd3aca494586a"
Sep 08 10:18:45.837278 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:45.837260    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b49ce63d5774293bfcb85f5506b009fd2bbad90874718eac9bd3aca494586a\": container with ID starting with 39b49ce63d5774293bfcb85f5506b009fd2bbad90874718eac9bd3aca494586a not found: ID does not exist" containerID="39b49ce63d5774293bfcb85f5506b009fd2bbad90874718eac9bd3aca494586a"
Sep 08 10:18:45.837331 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.837286    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b49ce63d5774293bfcb85f5506b009fd2bbad90874718eac9bd3aca494586a"} err="failed to get container status \"39b49ce63d5774293bfcb85f5506b009fd2bbad90874718eac9bd3aca494586a\": rpc error: code = NotFound desc = could not find container \"39b49ce63d5774293bfcb85f5506b009fd2bbad90874718eac9bd3aca494586a\": container with ID starting with 39b49ce63d5774293bfcb85f5506b009fd2bbad90874718eac9bd3aca494586a not found: ID does not exist"
Sep 08 10:18:45.837331 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.837303    2559 scope.go:117] "RemoveContainer" containerID="a6de3ca1d82ad2da102e4252a22846765b71a69e8adc4d797bcc0e1119ddcb98"
Sep 08 10:18:45.845200 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.845184    2559 scope.go:117] "RemoveContainer" containerID="a6de3ca1d82ad2da102e4252a22846765b71a69e8adc4d797bcc0e1119ddcb98"
Sep 08 10:18:45.845548 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:45.845531    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6de3ca1d82ad2da102e4252a22846765b71a69e8adc4d797bcc0e1119ddcb98\": container with ID starting with a6de3ca1d82ad2da102e4252a22846765b71a69e8adc4d797bcc0e1119ddcb98 not found: ID does not exist" containerID="a6de3ca1d82ad2da102e4252a22846765b71a69e8adc4d797bcc0e1119ddcb98"
Sep 08 10:18:45.845608 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.845562    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6de3ca1d82ad2da102e4252a22846765b71a69e8adc4d797bcc0e1119ddcb98"} err="failed to get container status \"a6de3ca1d82ad2da102e4252a22846765b71a69e8adc4d797bcc0e1119ddcb98\": rpc error: code = NotFound desc = could not find container \"a6de3ca1d82ad2da102e4252a22846765b71a69e8adc4d797bcc0e1119ddcb98\": container with ID starting with a6de3ca1d82ad2da102e4252a22846765b71a69e8adc4d797bcc0e1119ddcb98 not found: ID does not exist"
Sep 08 10:18:45.855981 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.855961    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"]
Sep 08 10:18:45.865868 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.865851    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c28013-7fdd4c9775-f6v2q"]
Sep 08 10:18:45.905174 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.905152    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"]
Sep 08 10:18:45.916149 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:45.916121    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c28013-f7477654-zkwh9"]
Sep 08 10:18:46.417718 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.417678    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn" podUID="46dbde84-2663-4cac-9d32-9627cdbd8288" containerName="main" containerID="cri-o://68652c866b16351498fdd60af06216a2a98974e632786a23fb3654b75ec6c224" gracePeriod=45
Sep 08 10:18:46.637266 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.637220    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0585fb9c-2d5e-48f8-a142-19cf206cbfa6" path="/var/lib/kubelet/pods/0585fb9c-2d5e-48f8-a142-19cf206cbfa6/volumes"
Sep 08 10:18:46.637621 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.637607    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d3559dc-7cce-42cd-9fd0-86afd305cd2e" path="/var/lib/kubelet/pods/3d3559dc-7cce-42cd-9fd0-86afd305cd2e/volumes"
Sep 08 10:18:46.679017 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.678967    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:18:46.715598 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.715574    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-home\") pod \"46dbde84-2663-4cac-9d32-9627cdbd8288\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") "
Sep 08 10:18:46.715715 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.715626    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg2fd\" (UniqueName: \"kubernetes.io/projected/46dbde84-2663-4cac-9d32-9627cdbd8288-kube-api-access-xg2fd\") pod \"46dbde84-2663-4cac-9d32-9627cdbd8288\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") "
Sep 08 10:18:46.715715 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.715648    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-tmp-dir\") pod \"46dbde84-2663-4cac-9d32-9627cdbd8288\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") "
Sep 08 10:18:46.715715 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.715681    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-dshm\") pod \"46dbde84-2663-4cac-9d32-9627cdbd8288\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") "
Sep 08 10:18:46.715715 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.715700    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46dbde84-2663-4cac-9d32-9627cdbd8288-tls-certs\") pod \"46dbde84-2663-4cac-9d32-9627cdbd8288\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") "
Sep 08 10:18:46.715927 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.715723    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-model-cache\") pod \"46dbde84-2663-4cac-9d32-9627cdbd8288\" (UID: \"46dbde84-2663-4cac-9d32-9627cdbd8288\") "
Sep 08 10:18:46.715927 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.715827    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-home" (OuterVolumeSpecName: "home") pod "46dbde84-2663-4cac-9d32-9627cdbd8288" (UID: "46dbde84-2663-4cac-9d32-9627cdbd8288"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:18:46.716038 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.715966    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "46dbde84-2663-4cac-9d32-9627cdbd8288" (UID: "46dbde84-2663-4cac-9d32-9627cdbd8288"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:18:46.716038 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.716022    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-model-cache" (OuterVolumeSpecName: "model-cache") pod "46dbde84-2663-4cac-9d32-9627cdbd8288" (UID: "46dbde84-2663-4cac-9d32-9627cdbd8288"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:18:46.716156 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.716055    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-home\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:46.717828 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.717803    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-dshm" (OuterVolumeSpecName: "dshm") pod "46dbde84-2663-4cac-9d32-9627cdbd8288" (UID: "46dbde84-2663-4cac-9d32-9627cdbd8288"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:18:46.717919 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.717863    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46dbde84-2663-4cac-9d32-9627cdbd8288-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46dbde84-2663-4cac-9d32-9627cdbd8288" (UID: "46dbde84-2663-4cac-9d32-9627cdbd8288"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:18:46.718153 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.718125    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46dbde84-2663-4cac-9d32-9627cdbd8288-kube-api-access-xg2fd" (OuterVolumeSpecName: "kube-api-access-xg2fd") pod "46dbde84-2663-4cac-9d32-9627cdbd8288" (UID: "46dbde84-2663-4cac-9d32-9627cdbd8288"). InnerVolumeSpecName "kube-api-access-xg2fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:18:46.816837 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.816802    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg2fd\" (UniqueName: \"kubernetes.io/projected/46dbde84-2663-4cac-9d32-9627cdbd8288-kube-api-access-xg2fd\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:46.816837 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.816835    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-tmp-dir\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:46.816837 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.816844    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-dshm\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:46.817003 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.816853    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46dbde84-2663-4cac-9d32-9627cdbd8288-tls-certs\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:46.817003 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.816860    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dbde84-2663-4cac-9d32-9627cdbd8288-model-cache\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:46.831694 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.831667    2559 generic.go:356] "Generic (PLEG): container finished" podID="46dbde84-2663-4cac-9d32-9627cdbd8288" containerID="68652c866b16351498fdd60af06216a2a98974e632786a23fb3654b75ec6c224" exitCode=0
Sep 08 10:18:46.831993 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.831715    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"
Sep 08 10:18:46.831993 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.831738    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn" event={"ID":"46dbde84-2663-4cac-9d32-9627cdbd8288","Type":"ContainerDied","Data":"68652c866b16351498fdd60af06216a2a98974e632786a23fb3654b75ec6c224"}
Sep 08 10:18:46.831993 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.831774    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn" event={"ID":"46dbde84-2663-4cac-9d32-9627cdbd8288","Type":"ContainerDied","Data":"e41b080dcd460267d75c253e21d936922c840a813576a33986af532b734c4467"}
Sep 08 10:18:46.831993 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.831796    2559 scope.go:117] "RemoveContainer" containerID="68652c866b16351498fdd60af06216a2a98974e632786a23fb3654b75ec6c224"
Sep 08 10:18:46.840640 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.840624    2559 scope.go:117] "RemoveContainer" containerID="68652c866b16351498fdd60af06216a2a98974e632786a23fb3654b75ec6c224"
Sep 08 10:18:46.840910 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:46.840895    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68652c866b16351498fdd60af06216a2a98974e632786a23fb3654b75ec6c224\": container with ID starting with 68652c866b16351498fdd60af06216a2a98974e632786a23fb3654b75ec6c224 not found: ID does not exist" containerID="68652c866b16351498fdd60af06216a2a98974e632786a23fb3654b75ec6c224"
Sep 08 10:18:46.840958 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.840918    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68652c866b16351498fdd60af06216a2a98974e632786a23fb3654b75ec6c224"} err="failed to get container status \"68652c866b16351498fdd60af06216a2a98974e632786a23fb3654b75ec6c224\": rpc error: code = NotFound desc = could not find container \"68652c866b16351498fdd60af06216a2a98974e632786a23fb3654b75ec6c224\": container with ID starting with 68652c866b16351498fdd60af06216a2a98974e632786a23fb3654b75ec6c224 not found: ID does not exist"
Sep 08 10:18:46.854467 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.854448    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"]
Sep 08 10:18:46.856325 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:46.856306    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c28013/test-model-c28013-kserve-7df7c86955-kkdqn"]
Sep 08 10:18:48.533015 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:48.532984    2559 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-self-signed-certs: secret "noten-model-738ee5-kserve-self-signed-certs" not found
Sep 08 10:18:48.533420 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:48.533065    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs podName:4cf02cae-a839-4125-aab2-a05efcd97c6b nodeName:}" failed. No retries permitted until 2026-09-08 10:19:04.533050977 +0000 UTC m=+1064.374275339 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs") pod "noten-model-738ee5-kserve-7b47788df9-jndf5" (UID: "4cf02cae-a839-4125-aab2-a05efcd97c6b") : secret "noten-model-738ee5-kserve-self-signed-certs" not found
Sep 08 10:18:48.636609 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:48.636576    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46dbde84-2663-4cac-9d32-9627cdbd8288" path="/var/lib/kubelet/pods/46dbde84-2663-4cac-9d32-9627cdbd8288/volumes"
Sep 08 10:18:48.771301 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:48.771258    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5" podUID="4cf02cae-a839-4125-aab2-a05efcd97c6b" containerName="main" containerID="cri-o://998796e74d3e45dd3f3b4d08666bc69448a694a85ca9a9e6838581c6bd310713" gracePeriod=45
Sep 08 10:18:49.029669 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.029646    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:49.137367 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.137293    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-home\") pod \"4cf02cae-a839-4125-aab2-a05efcd97c6b\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") "
Sep 08 10:18:49.137367 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.137324    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-dshm\") pod \"4cf02cae-a839-4125-aab2-a05efcd97c6b\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") "
Sep 08 10:18:49.137367 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.137354    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbzqb\" (UniqueName: \"kubernetes.io/projected/4cf02cae-a839-4125-aab2-a05efcd97c6b-kube-api-access-kbzqb\") pod \"4cf02cae-a839-4125-aab2-a05efcd97c6b\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") "
Sep 08 10:18:49.137695 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.137388    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs\") pod \"4cf02cae-a839-4125-aab2-a05efcd97c6b\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") "
Sep 08 10:18:49.137695 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.137470    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-model-cache\") pod \"4cf02cae-a839-4125-aab2-a05efcd97c6b\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") "
Sep 08 10:18:49.137695 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.137552    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-tmp-dir\") pod \"4cf02cae-a839-4125-aab2-a05efcd97c6b\" (UID: \"4cf02cae-a839-4125-aab2-a05efcd97c6b\") "
Sep 08 10:18:49.137695 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.137578    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-home" (OuterVolumeSpecName: "home") pod "4cf02cae-a839-4125-aab2-a05efcd97c6b" (UID: "4cf02cae-a839-4125-aab2-a05efcd97c6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:18:49.137909 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.137833    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-home\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:49.137909 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.137813    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "4cf02cae-a839-4125-aab2-a05efcd97c6b" (UID: "4cf02cae-a839-4125-aab2-a05efcd97c6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:18:49.137909 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.137858    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4cf02cae-a839-4125-aab2-a05efcd97c6b" (UID: "4cf02cae-a839-4125-aab2-a05efcd97c6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:18:49.139681 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.139658    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf02cae-a839-4125-aab2-a05efcd97c6b-kube-api-access-kbzqb" (OuterVolumeSpecName: "kube-api-access-kbzqb") pod "4cf02cae-a839-4125-aab2-a05efcd97c6b" (UID: "4cf02cae-a839-4125-aab2-a05efcd97c6b"). InnerVolumeSpecName "kube-api-access-kbzqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:18:49.139783 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.139693    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-dshm" (OuterVolumeSpecName: "dshm") pod "4cf02cae-a839-4125-aab2-a05efcd97c6b" (UID: "4cf02cae-a839-4125-aab2-a05efcd97c6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:18:49.139783 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.139707    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4cf02cae-a839-4125-aab2-a05efcd97c6b" (UID: "4cf02cae-a839-4125-aab2-a05efcd97c6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:18:49.239068 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.239040    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-dshm\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:49.239068 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.239068    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbzqb\" (UniqueName: \"kubernetes.io/projected/4cf02cae-a839-4125-aab2-a05efcd97c6b-kube-api-access-kbzqb\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:49.239204 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.239081    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf02cae-a839-4125-aab2-a05efcd97c6b-tls-certs\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:49.239204 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.239093    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-model-cache\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:49.239204 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.239104    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf02cae-a839-4125-aab2-a05efcd97c6b-tmp-dir\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:49.850416 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.850384    2559 generic.go:356] "Generic (PLEG): container finished" podID="4cf02cae-a839-4125-aab2-a05efcd97c6b" containerID="998796e74d3e45dd3f3b4d08666bc69448a694a85ca9a9e6838581c6bd310713" exitCode=0
Sep 08 10:18:49.850852 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.850427    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"
Sep 08 10:18:49.850852 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.850461    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5" event={"ID":"4cf02cae-a839-4125-aab2-a05efcd97c6b","Type":"ContainerDied","Data":"998796e74d3e45dd3f3b4d08666bc69448a694a85ca9a9e6838581c6bd310713"}
Sep 08 10:18:49.850852 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.850504    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5" event={"ID":"4cf02cae-a839-4125-aab2-a05efcd97c6b","Type":"ContainerDied","Data":"bebe2e572813cfac9c5fb74f636858c309db4ca10032073e529c6a11231794cc"}
Sep 08 10:18:49.850852 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.850520    2559 scope.go:117] "RemoveContainer" containerID="998796e74d3e45dd3f3b4d08666bc69448a694a85ca9a9e6838581c6bd310713"
Sep 08 10:18:49.865264 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.865230    2559 scope.go:117] "RemoveContainer" containerID="998796e74d3e45dd3f3b4d08666bc69448a694a85ca9a9e6838581c6bd310713"
Sep 08 10:18:49.865512 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:18:49.865494    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998796e74d3e45dd3f3b4d08666bc69448a694a85ca9a9e6838581c6bd310713\": container with ID starting with 998796e74d3e45dd3f3b4d08666bc69448a694a85ca9a9e6838581c6bd310713 not found: ID does not exist" containerID="998796e74d3e45dd3f3b4d08666bc69448a694a85ca9a9e6838581c6bd310713"
Sep 08 10:18:49.865568 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.865519    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998796e74d3e45dd3f3b4d08666bc69448a694a85ca9a9e6838581c6bd310713"} err="failed to get container status \"998796e74d3e45dd3f3b4d08666bc69448a694a85ca9a9e6838581c6bd310713\": rpc error: code = NotFound desc = could not find container \"998796e74d3e45dd3f3b4d08666bc69448a694a85ca9a9e6838581c6bd310713\": container with ID starting with 998796e74d3e45dd3f3b4d08666bc69448a694a85ca9a9e6838581c6bd310713 not found: ID does not exist"
Sep 08 10:18:49.880432 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.880410    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"]
Sep 08 10:18:49.885388 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:49.885366    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-738ee5/noten-model-738ee5-kserve-7b47788df9-jndf5"]
Sep 08 10:18:50.636137 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:50.636105    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cf02cae-a839-4125-aab2-a05efcd97c6b" path="/var/lib/kubelet/pods/4cf02cae-a839-4125-aab2-a05efcd97c6b/volumes"
Sep 08 10:18:52.802176 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.802137    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk"]
Sep 08 10:18:52.803025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.802703    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d3559dc-7cce-42cd-9fd0-86afd305cd2e" containerName="payload-pre-processing"
Sep 08 10:18:52.803025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.802724    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d3559dc-7cce-42cd-9fd0-86afd305cd2e" containerName="payload-pre-processing"
Sep 08 10:18:52.803025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.802740    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf02cae-a839-4125-aab2-a05efcd97c6b" containerName="main"
Sep 08 10:18:52.803025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.802748    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf02cae-a839-4125-aab2-a05efcd97c6b" containerName="main"
Sep 08 10:18:52.803025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.802774    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46dbde84-2663-4cac-9d32-9627cdbd8288" containerName="main"
Sep 08 10:18:52.803025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.802783    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="46dbde84-2663-4cac-9d32-9627cdbd8288" containerName="main"
Sep 08 10:18:52.803025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.802796    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0585fb9c-2d5e-48f8-a142-19cf206cbfa6" containerName="payload-processing"
Sep 08 10:18:52.803025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.802805    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0585fb9c-2d5e-48f8-a142-19cf206cbfa6" containerName="payload-processing"
Sep 08 10:18:52.803025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.802906    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf02cae-a839-4125-aab2-a05efcd97c6b" containerName="main"
Sep 08 10:18:52.803025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.802917    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0585fb9c-2d5e-48f8-a142-19cf206cbfa6" containerName="payload-processing"
Sep 08 10:18:52.803025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.802933    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d3559dc-7cce-42cd-9fd0-86afd305cd2e" containerName="payload-pre-processing"
Sep 08 10:18:52.803025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.802942    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="46dbde84-2663-4cac-9d32-9627cdbd8288" containerName="main"
Sep 08 10:18:52.807552 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.807529    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk"
Sep 08 10:18:52.811528 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.811508    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:18:52.811627 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.811511    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:18:52.819455 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.819435    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk"]
Sep 08 10:18:52.870989 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.870964    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e385fca-0606-4ed7-b5ea-f574a2846068-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk\" (UID: \"4e385fca-0606-4ed7-b5ea-f574a2846068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk"
Sep 08 10:18:52.972408 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.972374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e385fca-0606-4ed7-b5ea-f574a2846068-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk\" (UID: \"4e385fca-0606-4ed7-b5ea-f574a2846068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk"
Sep 08 10:18:52.973002 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:52.972981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e385fca-0606-4ed7-b5ea-f574a2846068-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk\" (UID: \"4e385fca-0606-4ed7-b5ea-f574a2846068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk"
Sep 08 10:18:53.117663 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:53.117589    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk"
Sep 08 10:18:53.405519 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:53.405496    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk"]
Sep 08 10:18:53.407682 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:18:53.407650    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e385fca_0606_4ed7_b5ea_f574a2846068.slice/crio-d3f1c9494302c301f7102f1e823aec1bfcdb2ac1a180ce4a7c71ec4cbdc2386a WatchSource:0}: Error finding container d3f1c9494302c301f7102f1e823aec1bfcdb2ac1a180ce4a7c71ec4cbdc2386a: Status 404 returned error can't find the container with id d3f1c9494302c301f7102f1e823aec1bfcdb2ac1a180ce4a7c71ec4cbdc2386a
Sep 08 10:18:53.875119 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:53.875085    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk" event={"ID":"4e385fca-0606-4ed7-b5ea-f574a2846068","Type":"ContainerStarted","Data":"3aca9214ace35cc95a29e9694d6efe03628d00e55f3d70876d330aa43e8c5297"}
Sep 08 10:18:53.875592 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:53.875136    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk" event={"ID":"4e385fca-0606-4ed7-b5ea-f574a2846068","Type":"ContainerStarted","Data":"d3f1c9494302c301f7102f1e823aec1bfcdb2ac1a180ce4a7c71ec4cbdc2386a"}
Sep 08 10:18:53.894787 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:53.894725    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk" podStartSLOduration=1.894709559 podStartE2EDuration="1.894709559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:18:53.8924588 +0000 UTC m=+1053.733683181" watchObservedRunningTime="2026-09-08 10:18:53.894709559 +0000 UTC m=+1053.735933942"
Sep 08 10:18:54.883303 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:54.883230    2559 generic.go:356] "Generic (PLEG): container finished" podID="4e385fca-0606-4ed7-b5ea-f574a2846068" containerID="3aca9214ace35cc95a29e9694d6efe03628d00e55f3d70876d330aa43e8c5297" exitCode=0
Sep 08 10:18:54.883706 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:54.883321    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk" event={"ID":"4e385fca-0606-4ed7-b5ea-f574a2846068","Type":"ContainerDied","Data":"3aca9214ace35cc95a29e9694d6efe03628d00e55f3d70876d330aa43e8c5297"}
Sep 08 10:18:56.016358 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:56.016334    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk"
Sep 08 10:18:56.103508 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:56.103477    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e385fca-0606-4ed7-b5ea-f574a2846068-maas-api-service-ca\") pod \"4e385fca-0606-4ed7-b5ea-f574a2846068\" (UID: \"4e385fca-0606-4ed7-b5ea-f574a2846068\") "
Sep 08 10:18:56.103736 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:56.103714    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e385fca-0606-4ed7-b5ea-f574a2846068-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e385fca-0606-4ed7-b5ea-f574a2846068" (UID: "4e385fca-0606-4ed7-b5ea-f574a2846068"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:18:56.103825 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:56.103814    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e385fca-0606-4ed7-b5ea-f574a2846068-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:18:56.895652 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:56.895574    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk"
Sep 08 10:18:56.895652 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:56.895604    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk" event={"ID":"4e385fca-0606-4ed7-b5ea-f574a2846068","Type":"ContainerDied","Data":"d3f1c9494302c301f7102f1e823aec1bfcdb2ac1a180ce4a7c71ec4cbdc2386a"}
Sep 08 10:18:56.895652 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:18:56.895645    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3f1c9494302c301f7102f1e823aec1bfcdb2ac1a180ce4a7c71ec4cbdc2386a"
Sep 08 10:19:03.447666 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.447629    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e365b-29814375-9kjsf"]
Sep 08 10:19:03.450479 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.450455    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e365b-29814375-9kjsf"]
Sep 08 10:19:03.481210 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.481184    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"]
Sep 08 10:19:03.481456 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.481432    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp" podUID="4121c456-ed92-4134-86ac-f6793aa35859" containerName="payload-processing" containerID="cri-o://54f103efd37fa8994a368038bf19fc928d18ac7cbdb42c2ee7bc000e94258327" gracePeriod=30
Sep 08 10:19:03.488086 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.488059    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"]
Sep 08 10:19:03.488347 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.488318    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z" podUID="e737b683-5474-48e4-954c-69b3ec23df8b" containerName="payload-pre-processing" containerID="cri-o://83b9537156bf6c31dbb16f507401d00c6a0511e55456faaa1aa257efb0d85cc1" gracePeriod=30
Sep 08 10:19:03.792507 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.792476    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"
Sep 08 10:19:03.796751 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.796733    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"
Sep 08 10:19:03.867503 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.867475    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk"]
Sep 08 10:19:03.868694 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.868653    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmwzl\" (UniqueName: \"kubernetes.io/projected/4121c456-ed92-4134-86ac-f6793aa35859-kube-api-access-kmwzl\") pod \"4121c456-ed92-4134-86ac-f6793aa35859\" (UID: \"4121c456-ed92-4134-86ac-f6793aa35859\") "
Sep 08 10:19:03.868836 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.868719    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e737b683-5474-48e4-954c-69b3ec23df8b-plugins-config-volume\") pod \"e737b683-5474-48e4-954c-69b3ec23df8b\" (UID: \"e737b683-5474-48e4-954c-69b3ec23df8b\") "
Sep 08 10:19:03.868836 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.868745    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtj5z\" (UniqueName: \"kubernetes.io/projected/e737b683-5474-48e4-954c-69b3ec23df8b-kube-api-access-mtj5z\") pod \"e737b683-5474-48e4-954c-69b3ec23df8b\" (UID: \"e737b683-5474-48e4-954c-69b3ec23df8b\") "
Sep 08 10:19:03.868836 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.868813    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4121c456-ed92-4134-86ac-f6793aa35859-plugins-config-volume\") pod \"4121c456-ed92-4134-86ac-f6793aa35859\" (UID: \"4121c456-ed92-4134-86ac-f6793aa35859\") "
Sep 08 10:19:03.869383 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.869218    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e737b683-5474-48e4-954c-69b3ec23df8b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e737b683-5474-48e4-954c-69b3ec23df8b" (UID: "e737b683-5474-48e4-954c-69b3ec23df8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:19:03.869486 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.869415    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4121c456-ed92-4134-86ac-f6793aa35859-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4121c456-ed92-4134-86ac-f6793aa35859" (UID: "4121c456-ed92-4134-86ac-f6793aa35859"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:19:03.870752 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.870734    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a3f5190-sqczk"]
Sep 08 10:19:03.870878 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.870858    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4121c456-ed92-4134-86ac-f6793aa35859-kube-api-access-kmwzl" (OuterVolumeSpecName: "kube-api-access-kmwzl") pod "4121c456-ed92-4134-86ac-f6793aa35859" (UID: "4121c456-ed92-4134-86ac-f6793aa35859"). InnerVolumeSpecName "kube-api-access-kmwzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:19:03.870989 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.870965    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e737b683-5474-48e4-954c-69b3ec23df8b-kube-api-access-mtj5z" (OuterVolumeSpecName: "kube-api-access-mtj5z") pod "e737b683-5474-48e4-954c-69b3ec23df8b" (UID: "e737b683-5474-48e4-954c-69b3ec23df8b"). InnerVolumeSpecName "kube-api-access-mtj5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:19:03.934859 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.934830    2559 generic.go:356] "Generic (PLEG): container finished" podID="4121c456-ed92-4134-86ac-f6793aa35859" containerID="54f103efd37fa8994a368038bf19fc928d18ac7cbdb42c2ee7bc000e94258327" exitCode=0
Sep 08 10:19:03.935003 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.934870    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp" event={"ID":"4121c456-ed92-4134-86ac-f6793aa35859","Type":"ContainerDied","Data":"54f103efd37fa8994a368038bf19fc928d18ac7cbdb42c2ee7bc000e94258327"}
Sep 08 10:19:03.935003 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.934881    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"
Sep 08 10:19:03.935003 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.934927    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp" event={"ID":"4121c456-ed92-4134-86ac-f6793aa35859","Type":"ContainerDied","Data":"88d09376dca2cc332dab4bb091cb4be5714bfb3c8295aec74ae79577778dfb3c"}
Sep 08 10:19:03.935003 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.934955    2559 scope.go:117] "RemoveContainer" containerID="54f103efd37fa8994a368038bf19fc928d18ac7cbdb42c2ee7bc000e94258327"
Sep 08 10:19:03.936616 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.936596    2559 generic.go:356] "Generic (PLEG): container finished" podID="e737b683-5474-48e4-954c-69b3ec23df8b" containerID="83b9537156bf6c31dbb16f507401d00c6a0511e55456faaa1aa257efb0d85cc1" exitCode=0
Sep 08 10:19:03.936729 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.936623    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z" event={"ID":"e737b683-5474-48e4-954c-69b3ec23df8b","Type":"ContainerDied","Data":"83b9537156bf6c31dbb16f507401d00c6a0511e55456faaa1aa257efb0d85cc1"}
Sep 08 10:19:03.936729 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.936643    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"
Sep 08 10:19:03.936729 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.936678    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z" event={"ID":"e737b683-5474-48e4-954c-69b3ec23df8b","Type":"ContainerDied","Data":"45aba2c287d523563b5eb356ebe72a2e22e845b104e0bb691f7659934e3fd37e"}
Sep 08 10:19:03.948845 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.948828    2559 scope.go:117] "RemoveContainer" containerID="54f103efd37fa8994a368038bf19fc928d18ac7cbdb42c2ee7bc000e94258327"
Sep 08 10:19:03.949101 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:19:03.949082    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f103efd37fa8994a368038bf19fc928d18ac7cbdb42c2ee7bc000e94258327\": container with ID starting with 54f103efd37fa8994a368038bf19fc928d18ac7cbdb42c2ee7bc000e94258327 not found: ID does not exist" containerID="54f103efd37fa8994a368038bf19fc928d18ac7cbdb42c2ee7bc000e94258327"
Sep 08 10:19:03.949158 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.949108    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f103efd37fa8994a368038bf19fc928d18ac7cbdb42c2ee7bc000e94258327"} err="failed to get container status \"54f103efd37fa8994a368038bf19fc928d18ac7cbdb42c2ee7bc000e94258327\": rpc error: code = NotFound desc = could not find container \"54f103efd37fa8994a368038bf19fc928d18ac7cbdb42c2ee7bc000e94258327\": container with ID starting with 54f103efd37fa8994a368038bf19fc928d18ac7cbdb42c2ee7bc000e94258327 not found: ID does not exist"
Sep 08 10:19:03.949158 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.949126    2559 scope.go:117] "RemoveContainer" containerID="83b9537156bf6c31dbb16f507401d00c6a0511e55456faaa1aa257efb0d85cc1"
Sep 08 10:19:03.958163 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.958127    2559 scope.go:117] "RemoveContainer" containerID="83b9537156bf6c31dbb16f507401d00c6a0511e55456faaa1aa257efb0d85cc1"
Sep 08 10:19:03.958542 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:19:03.958516    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b9537156bf6c31dbb16f507401d00c6a0511e55456faaa1aa257efb0d85cc1\": container with ID starting with 83b9537156bf6c31dbb16f507401d00c6a0511e55456faaa1aa257efb0d85cc1 not found: ID does not exist" containerID="83b9537156bf6c31dbb16f507401d00c6a0511e55456faaa1aa257efb0d85cc1"
Sep 08 10:19:03.958636 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.958552    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b9537156bf6c31dbb16f507401d00c6a0511e55456faaa1aa257efb0d85cc1"} err="failed to get container status \"83b9537156bf6c31dbb16f507401d00c6a0511e55456faaa1aa257efb0d85cc1\": rpc error: code = NotFound desc = could not find container \"83b9537156bf6c31dbb16f507401d00c6a0511e55456faaa1aa257efb0d85cc1\": container with ID starting with 83b9537156bf6c31dbb16f507401d00c6a0511e55456faaa1aa257efb0d85cc1 not found: ID does not exist"
Sep 08 10:19:03.959028 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.959006    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"]
Sep 08 10:19:03.963400 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.963380    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9e365b-dc4679495-fr6zp"]
Sep 08 10:19:03.970408 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.970392    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtj5z\" (UniqueName: \"kubernetes.io/projected/e737b683-5474-48e4-954c-69b3ec23df8b-kube-api-access-mtj5z\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:19:03.970408 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.970408    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4121c456-ed92-4134-86ac-f6793aa35859-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:19:03.970525 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.970417    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmwzl\" (UniqueName: \"kubernetes.io/projected/4121c456-ed92-4134-86ac-f6793aa35859-kube-api-access-kmwzl\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:19:03.970525 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.970426    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e737b683-5474-48e4-954c-69b3ec23df8b-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:19:03.979975 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.979952    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"]
Sep 08 10:19:03.986389 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:03.986370    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9e365b-6b5b96c7fd-52x7z"]
Sep 08 10:19:04.638217 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:04.638184    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4121c456-ed92-4134-86ac-f6793aa35859" path="/var/lib/kubelet/pods/4121c456-ed92-4134-86ac-f6793aa35859/volumes"
Sep 08 10:19:04.638568 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:04.638529    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e385fca-0606-4ed7-b5ea-f574a2846068" path="/var/lib/kubelet/pods/4e385fca-0606-4ed7-b5ea-f574a2846068/volumes"
Sep 08 10:19:04.638768 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:04.638755    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc5bda56-cf56-4f6d-8b35-1086108d7503" path="/var/lib/kubelet/pods/cc5bda56-cf56-4f6d-8b35-1086108d7503/volumes"
Sep 08 10:19:04.639031 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:04.639020    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e737b683-5474-48e4-954c-69b3ec23df8b" path="/var/lib/kubelet/pods/e737b683-5474-48e4-954c-69b3ec23df8b/volumes"
Sep 08 10:19:16.424464 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.424385    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"]
Sep 08 10:19:16.424881 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.424823    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4121c456-ed92-4134-86ac-f6793aa35859" containerName="payload-processing"
Sep 08 10:19:16.424881 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.424841    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4121c456-ed92-4134-86ac-f6793aa35859" containerName="payload-processing"
Sep 08 10:19:16.424881 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.424867    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e385fca-0606-4ed7-b5ea-f574a2846068" containerName="revoke-keys"
Sep 08 10:19:16.424881 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.424876    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e385fca-0606-4ed7-b5ea-f574a2846068" containerName="revoke-keys"
Sep 08 10:19:16.425099 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.424894    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e737b683-5474-48e4-954c-69b3ec23df8b" containerName="payload-pre-processing"
Sep 08 10:19:16.425099 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.424903    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e737b683-5474-48e4-954c-69b3ec23df8b" containerName="payload-pre-processing"
Sep 08 10:19:16.425099 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.424988    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e737b683-5474-48e4-954c-69b3ec23df8b" containerName="payload-pre-processing"
Sep 08 10:19:16.425099 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.425000    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4121c456-ed92-4134-86ac-f6793aa35859" containerName="payload-processing"
Sep 08 10:19:16.425099 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.425008    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e385fca-0606-4ed7-b5ea-f574a2846068" containerName="revoke-keys"
Sep 08 10:19:16.429774 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.429753    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"
Sep 08 10:19:16.432286 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.432239    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-71a7a1\""
Sep 08 10:19:16.432398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.432300    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-71a7a1-dockercfg-l882w\""
Sep 08 10:19:16.438905 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.438880    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"]
Sep 08 10:19:16.581199 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.581161    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb57f33-d29d-47c8-92cb-a293d73067fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g\" (UID: \"3eb57f33-d29d-47c8-92cb-a293d73067fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"
Sep 08 10:19:16.581376 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.581226    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jms6m\" (UniqueName: \"kubernetes.io/projected/3eb57f33-d29d-47c8-92cb-a293d73067fe-kube-api-access-jms6m\") pod \"payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g\" (UID: \"3eb57f33-d29d-47c8-92cb-a293d73067fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"
Sep 08 10:19:16.681868 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.681788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb57f33-d29d-47c8-92cb-a293d73067fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g\" (UID: \"3eb57f33-d29d-47c8-92cb-a293d73067fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"
Sep 08 10:19:16.681868 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.681854    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jms6m\" (UniqueName: \"kubernetes.io/projected/3eb57f33-d29d-47c8-92cb-a293d73067fe-kube-api-access-jms6m\") pod \"payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g\" (UID: \"3eb57f33-d29d-47c8-92cb-a293d73067fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"
Sep 08 10:19:16.682481 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.682460    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb57f33-d29d-47c8-92cb-a293d73067fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g\" (UID: \"3eb57f33-d29d-47c8-92cb-a293d73067fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"
Sep 08 10:19:16.692272 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.692236    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jms6m\" (UniqueName: \"kubernetes.io/projected/3eb57f33-d29d-47c8-92cb-a293d73067fe-kube-api-access-jms6m\") pod \"payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g\" (UID: \"3eb57f33-d29d-47c8-92cb-a293d73067fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"
Sep 08 10:19:16.741214 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:16.741193    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"
Sep 08 10:19:17.013040 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:17.013016    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"]
Sep 08 10:19:17.014276 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:19:17.014236    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb57f33_d29d_47c8_92cb_a293d73067fe.slice/crio-02076e6e5dcd077eee2c8151e9f3f55a96d89ae242b28d39bb357f784e9048c9 WatchSource:0}: Error finding container 02076e6e5dcd077eee2c8151e9f3f55a96d89ae242b28d39bb357f784e9048c9: Status 404 returned error can't find the container with id 02076e6e5dcd077eee2c8151e9f3f55a96d89ae242b28d39bb357f784e9048c9
Sep 08 10:19:18.015189 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:18.015152    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g" event={"ID":"3eb57f33-d29d-47c8-92cb-a293d73067fe","Type":"ContainerStarted","Data":"f4f0085ee5c1dbb638b176f336562bdfc122b35938817afb17f9fea9d0f82b20"}
Sep 08 10:19:18.015685 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:18.015213    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g" event={"ID":"3eb57f33-d29d-47c8-92cb-a293d73067fe","Type":"ContainerStarted","Data":"02076e6e5dcd077eee2c8151e9f3f55a96d89ae242b28d39bb357f784e9048c9"}
Sep 08 10:19:18.015685 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:18.015300    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"
Sep 08 10:19:18.047801 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:18.047755    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g" podStartSLOduration=2.047742738 podStartE2EDuration="2.047742738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:19:18.045845772 +0000 UTC m=+1077.887070155" watchObservedRunningTime="2026-09-08 10:19:18.047742738 +0000 UTC m=+1077.888967120"
Sep 08 10:19:29.023131 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:29.023063    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"
Sep 08 10:19:31.513258 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.513222    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"]
Sep 08 10:19:31.518109 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.518092    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"
Sep 08 10:19:31.520762 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.520739    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-af58b2\""
Sep 08 10:19:31.527736 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.527713    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"]
Sep 08 10:19:31.546063 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.546041    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"]
Sep 08 10:19:31.553547 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.550045    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"
Sep 08 10:19:31.554196 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.554079    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-af58b2-dockercfg-vzwqf\""
Sep 08 10:19:31.564440 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.564419    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"]
Sep 08 10:19:31.607557 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.607533    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7bf9bb-5ff5-4a80-bf42-3f588e913045-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g\" (UID: \"ed7bf9bb-5ff5-4a80-bf42-3f588e913045\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"
Sep 08 10:19:31.607651 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.607564    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd58x\" (UniqueName: \"kubernetes.io/projected/ed7bf9bb-5ff5-4a80-bf42-3f588e913045-kube-api-access-wd58x\") pod \"payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g\" (UID: \"ed7bf9bb-5ff5-4a80-bf42-3f588e913045\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"
Sep 08 10:19:31.607651 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.607602    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c84l8\" (UniqueName: \"kubernetes.io/projected/b1b75c89-93d4-42fe-b1ba-c035e219439a-kube-api-access-c84l8\") pod \"payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj\" (UID: \"b1b75c89-93d4-42fe-b1ba-c035e219439a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"
Sep 08 10:19:31.607729 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.607658    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b75c89-93d4-42fe-b1ba-c035e219439a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj\" (UID: \"b1b75c89-93d4-42fe-b1ba-c035e219439a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"
Sep 08 10:19:31.708742 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.708712    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7bf9bb-5ff5-4a80-bf42-3f588e913045-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g\" (UID: \"ed7bf9bb-5ff5-4a80-bf42-3f588e913045\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"
Sep 08 10:19:31.708863 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.708747    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd58x\" (UniqueName: \"kubernetes.io/projected/ed7bf9bb-5ff5-4a80-bf42-3f588e913045-kube-api-access-wd58x\") pod \"payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g\" (UID: \"ed7bf9bb-5ff5-4a80-bf42-3f588e913045\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"
Sep 08 10:19:31.708863 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.708781    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c84l8\" (UniqueName: \"kubernetes.io/projected/b1b75c89-93d4-42fe-b1ba-c035e219439a-kube-api-access-c84l8\") pod \"payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj\" (UID: \"b1b75c89-93d4-42fe-b1ba-c035e219439a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"
Sep 08 10:19:31.708863 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.708815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b75c89-93d4-42fe-b1ba-c035e219439a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj\" (UID: \"b1b75c89-93d4-42fe-b1ba-c035e219439a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"
Sep 08 10:19:31.709447 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.709429    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b75c89-93d4-42fe-b1ba-c035e219439a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj\" (UID: \"b1b75c89-93d4-42fe-b1ba-c035e219439a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"
Sep 08 10:19:31.709536 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.709520    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7bf9bb-5ff5-4a80-bf42-3f588e913045-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g\" (UID: \"ed7bf9bb-5ff5-4a80-bf42-3f588e913045\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"
Sep 08 10:19:31.717646 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.717620    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd58x\" (UniqueName: \"kubernetes.io/projected/ed7bf9bb-5ff5-4a80-bf42-3f588e913045-kube-api-access-wd58x\") pod \"payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g\" (UID: \"ed7bf9bb-5ff5-4a80-bf42-3f588e913045\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"
Sep 08 10:19:31.718063 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.718039    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c84l8\" (UniqueName: \"kubernetes.io/projected/b1b75c89-93d4-42fe-b1ba-c035e219439a-kube-api-access-c84l8\") pod \"payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj\" (UID: \"b1b75c89-93d4-42fe-b1ba-c035e219439a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"
Sep 08 10:19:31.828624 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.828563    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"
Sep 08 10:19:31.864358 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:31.864326    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"
Sep 08 10:19:32.109745 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:32.109723    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"]
Sep 08 10:19:32.111643 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:19:32.111619    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7bf9bb_5ff5_4a80_bf42_3f588e913045.slice/crio-c2477a96154d63da534e168ccf6a70ece78ed75c2d306ea4202fb7a7abc7e9cf WatchSource:0}: Error finding container c2477a96154d63da534e168ccf6a70ece78ed75c2d306ea4202fb7a7abc7e9cf: Status 404 returned error can't find the container with id c2477a96154d63da534e168ccf6a70ece78ed75c2d306ea4202fb7a7abc7e9cf
Sep 08 10:19:32.145227 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:32.145208    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"]
Sep 08 10:19:32.150181 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:19:32.150162    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b75c89_93d4_42fe_b1ba_c035e219439a.slice/crio-ffaeab56982a630b6e0059ba47b3bea6386420abf251f777ca606db90d2ccd35 WatchSource:0}: Error finding container ffaeab56982a630b6e0059ba47b3bea6386420abf251f777ca606db90d2ccd35: Status 404 returned error can't find the container with id ffaeab56982a630b6e0059ba47b3bea6386420abf251f777ca606db90d2ccd35
Sep 08 10:19:33.100413 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:33.100370    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj" event={"ID":"b1b75c89-93d4-42fe-b1ba-c035e219439a","Type":"ContainerStarted","Data":"32f7e2fc987afe337f99d4a2e1b3566cbcdd81946c34716e93e627b148ee7b9e"}
Sep 08 10:19:33.100836 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:33.100545    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"
Sep 08 10:19:33.100836 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:33.100595    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj" event={"ID":"b1b75c89-93d4-42fe-b1ba-c035e219439a","Type":"ContainerStarted","Data":"ffaeab56982a630b6e0059ba47b3bea6386420abf251f777ca606db90d2ccd35"}
Sep 08 10:19:33.101991 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:33.101967    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g" event={"ID":"ed7bf9bb-5ff5-4a80-bf42-3f588e913045","Type":"ContainerStarted","Data":"2884024e0751ea345b660347dc6ff47552e6b62f49b04337cb75bbc41949922c"}
Sep 08 10:19:33.102119 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:33.102013    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"
Sep 08 10:19:33.102119 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:33.102035    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g" event={"ID":"ed7bf9bb-5ff5-4a80-bf42-3f588e913045","Type":"ContainerStarted","Data":"c2477a96154d63da534e168ccf6a70ece78ed75c2d306ea4202fb7a7abc7e9cf"}
Sep 08 10:19:33.120418 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:33.120375    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj" podStartSLOduration=2.120363048 podStartE2EDuration="2.120363048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:19:33.118429285 +0000 UTC m=+1092.959653698" watchObservedRunningTime="2026-09-08 10:19:33.120363048 +0000 UTC m=+1092.961587430"
Sep 08 10:19:33.135934 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:33.135897    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g" podStartSLOduration=2.135885175 podStartE2EDuration="2.135885175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:19:33.134953064 +0000 UTC m=+1092.976177440" watchObservedRunningTime="2026-09-08 10:19:33.135885175 +0000 UTC m=+1092.977109558"
Sep 08 10:19:44.108837 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:44.108808    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"
Sep 08 10:19:44.109300 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:44.108870    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"
Sep 08 10:19:45.577830 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.577799    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"]
Sep 08 10:19:45.581462 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.581444    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"
Sep 08 10:19:45.584273 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.584236    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d6e53c\""
Sep 08 10:19:45.584371 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.584284    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d6e53c-dockercfg-28pnb\""
Sep 08 10:19:45.596051 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.596022    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"]
Sep 08 10:19:45.628950 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.628926    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc275459-5945-43bf-9575-8a20739f5ce6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd\" (UID: \"fc275459-5945-43bf-9575-8a20739f5ce6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"
Sep 08 10:19:45.629055 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.629014    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bwm4\" (UniqueName: \"kubernetes.io/projected/fc275459-5945-43bf-9575-8a20739f5ce6-kube-api-access-8bwm4\") pod \"payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd\" (UID: \"fc275459-5945-43bf-9575-8a20739f5ce6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"
Sep 08 10:19:45.641560 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.641536    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"]
Sep 08 10:19:45.644865 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.644850    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"
Sep 08 10:19:45.658683 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.658662    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"]
Sep 08 10:19:45.729995 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.729966    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rmvg\" (UniqueName: \"kubernetes.io/projected/98c49713-ae56-4458-97f9-581561fe431e-kube-api-access-7rmvg\") pod \"payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs\" (UID: \"98c49713-ae56-4458-97f9-581561fe431e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"
Sep 08 10:19:45.730152 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.730070    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bwm4\" (UniqueName: \"kubernetes.io/projected/fc275459-5945-43bf-9575-8a20739f5ce6-kube-api-access-8bwm4\") pod \"payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd\" (UID: \"fc275459-5945-43bf-9575-8a20739f5ce6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"
Sep 08 10:19:45.730152 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.730143    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c49713-ae56-4458-97f9-581561fe431e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs\" (UID: \"98c49713-ae56-4458-97f9-581561fe431e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"
Sep 08 10:19:45.730315 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.730234    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc275459-5945-43bf-9575-8a20739f5ce6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd\" (UID: \"fc275459-5945-43bf-9575-8a20739f5ce6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"
Sep 08 10:19:45.730893 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.730872    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc275459-5945-43bf-9575-8a20739f5ce6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd\" (UID: \"fc275459-5945-43bf-9575-8a20739f5ce6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"
Sep 08 10:19:45.739019 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.738995    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bwm4\" (UniqueName: \"kubernetes.io/projected/fc275459-5945-43bf-9575-8a20739f5ce6-kube-api-access-8bwm4\") pod \"payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd\" (UID: \"fc275459-5945-43bf-9575-8a20739f5ce6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"
Sep 08 10:19:45.831069 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.831017    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rmvg\" (UniqueName: \"kubernetes.io/projected/98c49713-ae56-4458-97f9-581561fe431e-kube-api-access-7rmvg\") pod \"payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs\" (UID: \"98c49713-ae56-4458-97f9-581561fe431e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"
Sep 08 10:19:45.831069 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.831053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c49713-ae56-4458-97f9-581561fe431e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs\" (UID: \"98c49713-ae56-4458-97f9-581561fe431e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"
Sep 08 10:19:45.831682 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.831665    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c49713-ae56-4458-97f9-581561fe431e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs\" (UID: \"98c49713-ae56-4458-97f9-581561fe431e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"
Sep 08 10:19:45.844389 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.844370    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rmvg\" (UniqueName: \"kubernetes.io/projected/98c49713-ae56-4458-97f9-581561fe431e-kube-api-access-7rmvg\") pod \"payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs\" (UID: \"98c49713-ae56-4458-97f9-581561fe431e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"
Sep 08 10:19:45.892658 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.892637    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"
Sep 08 10:19:45.954628 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:45.954594    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"
Sep 08 10:19:46.170869 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:46.170822    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"]
Sep 08 10:19:46.172220 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:19:46.172194    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc275459_5945_43bf_9575_8a20739f5ce6.slice/crio-bc2d24cd9350f1208618cca15a4d00c2f4718dba33f113cc420ef76cfe429a3e WatchSource:0}: Error finding container bc2d24cd9350f1208618cca15a4d00c2f4718dba33f113cc420ef76cfe429a3e: Status 404 returned error can't find the container with id bc2d24cd9350f1208618cca15a4d00c2f4718dba33f113cc420ef76cfe429a3e
Sep 08 10:19:46.242135 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:19:46.242114    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c49713_ae56_4458_97f9_581561fe431e.slice/crio-76f2060080de53eee36f76139ae7a4e235707fc463c42e76f3ec433803b2bf4c WatchSource:0}: Error finding container 76f2060080de53eee36f76139ae7a4e235707fc463c42e76f3ec433803b2bf4c: Status 404 returned error can't find the container with id 76f2060080de53eee36f76139ae7a4e235707fc463c42e76f3ec433803b2bf4c
Sep 08 10:19:46.242235 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:46.242159    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"]
Sep 08 10:19:47.191571 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:47.191531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs" event={"ID":"98c49713-ae56-4458-97f9-581561fe431e","Type":"ContainerStarted","Data":"d7758752e522e0d4e7ea9b63fda6717eeea7bd068a6f040fafd73b4eab67db45"}
Sep 08 10:19:47.192028 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:47.191647    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"
Sep 08 10:19:47.192028 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:47.191735    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs" event={"ID":"98c49713-ae56-4458-97f9-581561fe431e","Type":"ContainerStarted","Data":"76f2060080de53eee36f76139ae7a4e235707fc463c42e76f3ec433803b2bf4c"}
Sep 08 10:19:47.193263 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:47.193217    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd" event={"ID":"fc275459-5945-43bf-9575-8a20739f5ce6","Type":"ContainerStarted","Data":"747cd23ce6a1dbff0e3b5ce96f019b393765d5e2aa74df97db8d2f60d5f047a9"}
Sep 08 10:19:47.193407 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:47.193295    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"
Sep 08 10:19:47.193407 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:47.193320    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd" event={"ID":"fc275459-5945-43bf-9575-8a20739f5ce6","Type":"ContainerStarted","Data":"bc2d24cd9350f1208618cca15a4d00c2f4718dba33f113cc420ef76cfe429a3e"}
Sep 08 10:19:47.259724 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:47.259663    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd" podStartSLOduration=2.259643372 podStartE2EDuration="2.259643372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:19:47.25866761 +0000 UTC m=+1107.099892017" watchObservedRunningTime="2026-09-08 10:19:47.259643372 +0000 UTC m=+1107.100867751"
Sep 08 10:19:47.260198 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:47.260164    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs" podStartSLOduration=2.260154587 podStartE2EDuration="2.260154587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:19:47.231549455 +0000 UTC m=+1107.072773858" watchObservedRunningTime="2026-09-08 10:19:47.260154587 +0000 UTC m=+1107.101378972"
Sep 08 10:19:58.201433 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:58.201403    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"
Sep 08 10:19:58.201880 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:19:58.201799    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"
Sep 08 10:20:00.978804 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:00.978772    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"]
Sep 08 10:20:00.982690 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:00.982668    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"
Sep 08 10:20:00.985743 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:00.985719    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-37ed05-dockercfg-msj4r\""
Sep 08 10:20:00.985853 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:00.985835    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-37ed05\""
Sep 08 10:20:00.997678 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:00.997657    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"]
Sep 08 10:20:01.064403 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:01.064374    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8e0fa7-f53d-400e-abcf-2b9f95fc0892-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj\" (UID: \"9d8e0fa7-f53d-400e-abcf-2b9f95fc0892\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"
Sep 08 10:20:01.064550 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:01.064431    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hz4p\" (UniqueName: \"kubernetes.io/projected/9d8e0fa7-f53d-400e-abcf-2b9f95fc0892-kube-api-access-9hz4p\") pod \"payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj\" (UID: \"9d8e0fa7-f53d-400e-abcf-2b9f95fc0892\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"
Sep 08 10:20:01.165602 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:01.165577    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8e0fa7-f53d-400e-abcf-2b9f95fc0892-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj\" (UID: \"9d8e0fa7-f53d-400e-abcf-2b9f95fc0892\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"
Sep 08 10:20:01.165748 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:01.165633    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hz4p\" (UniqueName: \"kubernetes.io/projected/9d8e0fa7-f53d-400e-abcf-2b9f95fc0892-kube-api-access-9hz4p\") pod \"payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj\" (UID: \"9d8e0fa7-f53d-400e-abcf-2b9f95fc0892\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"
Sep 08 10:20:01.166259 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:01.166228    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8e0fa7-f53d-400e-abcf-2b9f95fc0892-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj\" (UID: \"9d8e0fa7-f53d-400e-abcf-2b9f95fc0892\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"
Sep 08 10:20:01.174526 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:01.174508    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hz4p\" (UniqueName: \"kubernetes.io/projected/9d8e0fa7-f53d-400e-abcf-2b9f95fc0892-kube-api-access-9hz4p\") pod \"payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj\" (UID: \"9d8e0fa7-f53d-400e-abcf-2b9f95fc0892\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"
Sep 08 10:20:01.293357 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:01.293298    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"
Sep 08 10:20:01.566015 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:01.565988    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"]
Sep 08 10:20:01.567300 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:20:01.567271    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8e0fa7_f53d_400e_abcf_2b9f95fc0892.slice/crio-d353ebf06175dbd7c01ee236beef83c7dd713515539a4053780e2dbff39335fa WatchSource:0}: Error finding container d353ebf06175dbd7c01ee236beef83c7dd713515539a4053780e2dbff39335fa: Status 404 returned error can't find the container with id d353ebf06175dbd7c01ee236beef83c7dd713515539a4053780e2dbff39335fa
Sep 08 10:20:02.277964 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:02.277930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj" event={"ID":"9d8e0fa7-f53d-400e-abcf-2b9f95fc0892","Type":"ContainerStarted","Data":"0487aa3597290fab55bf5a415fe2a7b13549202656afb09cc72f09b46b796792"}
Sep 08 10:20:02.278366 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:02.277995    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"
Sep 08 10:20:02.278366 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:02.278019    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj" event={"ID":"9d8e0fa7-f53d-400e-abcf-2b9f95fc0892","Type":"ContainerStarted","Data":"d353ebf06175dbd7c01ee236beef83c7dd713515539a4053780e2dbff39335fa"}
Sep 08 10:20:02.300357 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:02.300309    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj" podStartSLOduration=2.300290345 podStartE2EDuration="2.300290345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:20:02.298639232 +0000 UTC m=+1122.139863615" watchObservedRunningTime="2026-09-08 10:20:02.300290345 +0000 UTC m=+1122.141514728"
Sep 08 10:20:12.847501 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.847470    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"]
Sep 08 10:20:12.852293 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.852271    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.855847 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.855826    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-71a7a1\"/\"default-dockercfg-rf8zh\""
Sep 08 10:20:12.855847 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.855830    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-71a7a1\"/\"kube-root-ca.crt\""
Sep 08 10:20:12.856041 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.855856    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-71a7a1\"/\"openshift-service-ca.crt\""
Sep 08 10:20:12.856041 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.855892    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-71a7a1\"/\"auth-test-model-71a7a1-kserve-self-signed-certs\""
Sep 08 10:20:12.862609 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.862586    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"]
Sep 08 10:20:12.867618 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.867597    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-home\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.867718 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.867624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-model-cache\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.867718 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.867652    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-dshm\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.867718 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.867669    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q62mv\" (UniqueName: \"kubernetes.io/projected/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-kube-api-access-q62mv\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.867839 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.867729    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-tls-certs\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.867839 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.867784    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-tmp-dir\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.968518 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.968486    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-home\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.968666 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.968531    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-model-cache\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.968666 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.968570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-dshm\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.968666 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.968594    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q62mv\" (UniqueName: \"kubernetes.io/projected/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-kube-api-access-q62mv\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.968666 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.968629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-tls-certs\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.968886 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.968701    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-tmp-dir\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.968943 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.968890    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-home\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.969000 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.968948    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-model-cache\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.969101 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.969082    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-tmp-dir\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.970839 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.970814    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-dshm\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.971216 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.971194    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-tls-certs\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:12.987953 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:12.987925    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q62mv\" (UniqueName: \"kubernetes.io/projected/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-kube-api-access-q62mv\") pod \"auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:13.162956 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:13.162927    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:13.285611 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:13.285583    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"
Sep 08 10:20:13.435194 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:13.435172    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"]
Sep 08 10:20:13.437845 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:20:13.437820    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e8158d3_d44b_4bdc_aae6_0125b02b69fb.slice/crio-4980c63213345f1e20ed8241c9c27a2d83ea3fc7185adc888f7d71a9d2bdee8e WatchSource:0}: Error finding container 4980c63213345f1e20ed8241c9c27a2d83ea3fc7185adc888f7d71a9d2bdee8e: Status 404 returned error can't find the container with id 4980c63213345f1e20ed8241c9c27a2d83ea3fc7185adc888f7d71a9d2bdee8e
Sep 08 10:20:14.354436 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:14.354403    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m" event={"ID":"8e8158d3-d44b-4bdc-aae6-0125b02b69fb","Type":"ContainerStarted","Data":"00539c6a95e680e3b9f961fd8915c6937c6a16f80b377531e47335bbe03e7cbd"}
Sep 08 10:20:14.354436 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:14.354442    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m" event={"ID":"8e8158d3-d44b-4bdc-aae6-0125b02b69fb","Type":"ContainerStarted","Data":"4980c63213345f1e20ed8241c9c27a2d83ea3fc7185adc888f7d71a9d2bdee8e"}
Sep 08 10:20:14.372722 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:14.372676    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m" podStartSLOduration=2.372662159 podStartE2EDuration="2.372662159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:20:14.370420186 +0000 UTC m=+1134.211644590" watchObservedRunningTime="2026-09-08 10:20:14.372662159 +0000 UTC m=+1134.213886541"
Sep 08 10:20:23.163604 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:23.163569    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:23.164059 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:23.163624    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:23.176193 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:23.176170    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:23.419587 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:23.419507    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:20:41.935842 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:41.935812    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"]
Sep 08 10:20:41.939465 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:41.939443    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"
Sep 08 10:20:41.941891 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:41.941870    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8118c657\""
Sep 08 10:20:41.942002 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:41.941982    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8118c657-dockercfg-5s5nj\""
Sep 08 10:20:41.950964 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:41.950942    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"]
Sep 08 10:20:42.026640 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:42.026611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpxnq\" (UniqueName: \"kubernetes.io/projected/22fa1f43-aff8-4f59-b6d7-e053a0fc0d22-kube-api-access-rpxnq\") pod \"payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd\" (UID: \"22fa1f43-aff8-4f59-b6d7-e053a0fc0d22\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"
Sep 08 10:20:42.026768 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:42.026696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fa1f43-aff8-4f59-b6d7-e053a0fc0d22-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd\" (UID: \"22fa1f43-aff8-4f59-b6d7-e053a0fc0d22\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"
Sep 08 10:20:42.128064 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:42.128031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fa1f43-aff8-4f59-b6d7-e053a0fc0d22-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd\" (UID: \"22fa1f43-aff8-4f59-b6d7-e053a0fc0d22\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"
Sep 08 10:20:42.128178 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:42.128095    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpxnq\" (UniqueName: \"kubernetes.io/projected/22fa1f43-aff8-4f59-b6d7-e053a0fc0d22-kube-api-access-rpxnq\") pod \"payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd\" (UID: \"22fa1f43-aff8-4f59-b6d7-e053a0fc0d22\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"
Sep 08 10:20:42.128740 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:42.128720    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fa1f43-aff8-4f59-b6d7-e053a0fc0d22-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd\" (UID: \"22fa1f43-aff8-4f59-b6d7-e053a0fc0d22\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"
Sep 08 10:20:42.136780 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:42.136756    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpxnq\" (UniqueName: \"kubernetes.io/projected/22fa1f43-aff8-4f59-b6d7-e053a0fc0d22-kube-api-access-rpxnq\") pod \"payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd\" (UID: \"22fa1f43-aff8-4f59-b6d7-e053a0fc0d22\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"
Sep 08 10:20:42.249733 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:42.249674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"
Sep 08 10:20:42.523701 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:42.523491    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"]
Sep 08 10:20:42.526624 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:20:42.526595    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fa1f43_aff8_4f59_b6d7_e053a0fc0d22.slice/crio-dfae41dc98f07da4c725de5dd212db581c72ae8b7012f512c30f9461f6383ae0 WatchSource:0}: Error finding container dfae41dc98f07da4c725de5dd212db581c72ae8b7012f512c30f9461f6383ae0: Status 404 returned error can't find the container with id dfae41dc98f07da4c725de5dd212db581c72ae8b7012f512c30f9461f6383ae0
Sep 08 10:20:43.521296 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:43.521198    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd" event={"ID":"22fa1f43-aff8-4f59-b6d7-e053a0fc0d22","Type":"ContainerStarted","Data":"acc10ba5490fa1595f698e5e6dd15a2ad237eb14e97bc7df552954e222ac0e50"}
Sep 08 10:20:43.521296 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:43.521290    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"
Sep 08 10:20:43.521685 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:43.521320    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd" event={"ID":"22fa1f43-aff8-4f59-b6d7-e053a0fc0d22","Type":"ContainerStarted","Data":"dfae41dc98f07da4c725de5dd212db581c72ae8b7012f512c30f9461f6383ae0"}
Sep 08 10:20:43.540196 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:43.540145    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd" podStartSLOduration=2.540130102 podStartE2EDuration="2.540130102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:20:43.538520082 +0000 UTC m=+1163.379744493" watchObservedRunningTime="2026-09-08 10:20:43.540130102 +0000 UTC m=+1163.381354485"
Sep 08 10:20:54.528723 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:54.528689    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"
Sep 08 10:20:55.866015 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:55.865975    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"]
Sep 08 10:20:55.869952 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:55.869927    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:55.872499 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:55.872459    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d6e53c\"/\"openshift-service-ca.crt\""
Sep 08 10:20:55.873348 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:55.873326    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d6e53c\"/\"kube-root-ca.crt\""
Sep 08 10:20:55.873348 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:55.873344    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d6e53c\"/\"auth-test-model-d6e53c-kserve-self-signed-certs\""
Sep 08 10:20:55.873516 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:55.873333    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d6e53c\"/\"default-dockercfg-spwd4\""
Sep 08 10:20:55.880566 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:55.880548    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"]
Sep 08 10:20:56.047652 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.047618    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-home\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.047652 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.047653    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-model-cache\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.047850 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.047678    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-tmp-dir\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.047850 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.047784    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd96c96c-8870-4020-9b8b-06f15575c819-tls-certs\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.047850 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.047814    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnz7w\" (UniqueName: \"kubernetes.io/projected/fd96c96c-8870-4020-9b8b-06f15575c819-kube-api-access-dnz7w\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.047950 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.047888    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-dshm\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.148465 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.148400    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-dshm\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.148465 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.148439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-home\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.148465 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.148459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-model-cache\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.148668 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.148510    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-tmp-dir\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.148668 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.148570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd96c96c-8870-4020-9b8b-06f15575c819-tls-certs\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.148668 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.148591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnz7w\" (UniqueName: \"kubernetes.io/projected/fd96c96c-8870-4020-9b8b-06f15575c819-kube-api-access-dnz7w\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.148951 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.148931    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-model-cache\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.148951 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.148948    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-home\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.149067 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.148988    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-tmp-dir\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.150647 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.150626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-dshm\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.150938 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.150916    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd96c96c-8870-4020-9b8b-06f15575c819-tls-certs\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.157709 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.157686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnz7w\" (UniqueName: \"kubernetes.io/projected/fd96c96c-8870-4020-9b8b-06f15575c819-kube-api-access-dnz7w\") pod \"auth-test-model-d6e53c-kserve-5bdf858747-9tngn\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.180575 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.180553    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:20:56.454187 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.454165    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"]
Sep 08 10:20:56.456143 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:20:56.456105    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd96c96c_8870_4020_9b8b_06f15575c819.slice/crio-de1abc0c1f587420d26f2414f677eacec504daa64ae3a5ac491f4cdd7130bcbb WatchSource:0}: Error finding container de1abc0c1f587420d26f2414f677eacec504daa64ae3a5ac491f4cdd7130bcbb: Status 404 returned error can't find the container with id de1abc0c1f587420d26f2414f677eacec504daa64ae3a5ac491f4cdd7130bcbb
Sep 08 10:20:56.597302 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.597264    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn" event={"ID":"fd96c96c-8870-4020-9b8b-06f15575c819","Type":"ContainerStarted","Data":"ee6b3fe59974131cf3ba968db291d2124afcae9fa10f24f5d76d98a6b8ccdc70"}
Sep 08 10:20:56.597470 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.597310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn" event={"ID":"fd96c96c-8870-4020-9b8b-06f15575c819","Type":"ContainerStarted","Data":"de1abc0c1f587420d26f2414f677eacec504daa64ae3a5ac491f4cdd7130bcbb"}
Sep 08 10:20:56.622890 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:20:56.622838    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn" podStartSLOduration=1.622823773 podStartE2EDuration="1.622823773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:20:56.620867993 +0000 UTC m=+1176.462092395" watchObservedRunningTime="2026-09-08 10:20:56.622823773 +0000 UTC m=+1176.464048201"
Sep 08 10:21:06.181105 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:06.181067    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:21:06.181612 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:06.181330    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:21:06.193987 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:06.193962    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:21:06.672797 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:06.672772    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:21:20.691844 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:20.691818    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:21:20.693671 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:20.693649    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:21:20.774945 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:20.774919    2559 scope.go:117] "RemoveContainer" containerID="3cbe99bcb8723c223d95dce8e22560f8050968b1feb29614e53e0b22ce6c6c5e"
Sep 08 10:21:20.783379 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:20.783358    2559 scope.go:117] "RemoveContainer" containerID="71fbf80a789d2932de835544fa61eae96c56af27f41e77946a4f621a8dc51743"
Sep 08 10:21:20.790980 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:20.790962    2559 scope.go:117] "RemoveContainer" containerID="6b930fa8e5627491fb609bd8e5a4026e929c72515aee92789e0f1da84b0912a9"
Sep 08 10:21:20.798467 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:20.798450    2559 scope.go:117] "RemoveContainer" containerID="b8e877b9ca76594a6ff2f01b92eb2f045e7dcdac4bcbe28abf7614e13fb80872"
Sep 08 10:21:24.036662 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.036621    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"]
Sep 08 10:21:24.042443 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.042418    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.046138 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.046062    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8118c657\"/\"default-dockercfg-r86hr\""
Sep 08 10:21:24.046308 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.046176    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8118c657\"/\"openshift-service-ca.crt\""
Sep 08 10:21:24.046563 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.046542    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8118c657\"/\"kube-root-ca.crt\""
Sep 08 10:21:24.046901 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.046606    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8118c657\"/\"e2e-lifecycle-model-8118c657-kserve-self-signed-certs\""
Sep 08 10:21:24.048216 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.048193    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"]
Sep 08 10:21:24.097712 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.097684    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjjrf\" (UniqueName: \"kubernetes.io/projected/96a7b577-5e8d-410d-98b9-08118481123b-kube-api-access-cjjrf\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.097828 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.097721    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a7b577-5e8d-410d-98b9-08118481123b-tls-certs\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.097828 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.097747    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-model-cache\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.097902 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.097848    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-tmp-dir\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.097902 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.097870    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-dshm\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.097966 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.097911    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-home\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.199133 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.199103    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-tmp-dir\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.199273 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.199136    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-dshm\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.199273 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.199169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-home\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.199273 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.199207    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjjrf\" (UniqueName: \"kubernetes.io/projected/96a7b577-5e8d-410d-98b9-08118481123b-kube-api-access-cjjrf\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.199273 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.199232    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a7b577-5e8d-410d-98b9-08118481123b-tls-certs\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.199491 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.199293    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-model-cache\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.199618 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.199598    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-tmp-dir\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.199712 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.199692    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-model-cache\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.199749 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.199697    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-home\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.201423 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.201403    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-dshm\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.201726 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.201708    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a7b577-5e8d-410d-98b9-08118481123b-tls-certs\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.209920 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.209898    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjjrf\" (UniqueName: \"kubernetes.io/projected/96a7b577-5e8d-410d-98b9-08118481123b-kube-api-access-cjjrf\") pod \"e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") " pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.354377 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.354317    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:24.628734 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.628704    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"]
Sep 08 10:21:24.629301 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:21:24.629279    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96a7b577_5e8d_410d_98b9_08118481123b.slice/crio-c6eb34035ce9bfb98d1da8de4cfb0b25c828e6f90a6f4a5ee4e6089e7d1567e2 WatchSource:0}: Error finding container c6eb34035ce9bfb98d1da8de4cfb0b25c828e6f90a6f4a5ee4e6089e7d1567e2: Status 404 returned error can't find the container with id c6eb34035ce9bfb98d1da8de4cfb0b25c828e6f90a6f4a5ee4e6089e7d1567e2
Sep 08 10:21:24.770677 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.770640    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb" event={"ID":"96a7b577-5e8d-410d-98b9-08118481123b","Type":"ContainerStarted","Data":"4891b59ca498c33ac69f395035c83c67e2f6ef156e9e4583e8b127606a2953c3"}
Sep 08 10:21:24.770677 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.770686    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb" event={"ID":"96a7b577-5e8d-410d-98b9-08118481123b","Type":"ContainerStarted","Data":"c6eb34035ce9bfb98d1da8de4cfb0b25c828e6f90a6f4a5ee4e6089e7d1567e2"}
Sep 08 10:21:24.789701 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:24.789654    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb" podStartSLOduration=0.789640234 podStartE2EDuration="789.640234ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:21:24.788043738 +0000 UTC m=+1204.629268142" watchObservedRunningTime="2026-09-08 10:21:24.789640234 +0000 UTC m=+1204.630864614"
Sep 08 10:21:34.355237 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:34.355203    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:34.355738 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:34.355340    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:34.368086 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:34.368064    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:21:34.835733 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:21:34.835700    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:22:00.073196 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:00.073161    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc"]
Sep 08 10:22:00.076714 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:00.076697    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc"
Sep 08 10:22:00.079173 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:00.079151    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:22:00.079335 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:00.079194    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:22:00.084066 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:00.084040    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc"]
Sep 08 10:22:00.118742 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:00.118718    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8feb759-1de2-4816-93bc-a80445233f5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8118c657-bj8cc\" (UID: \"f8feb759-1de2-4816-93bc-a80445233f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc"
Sep 08 10:22:00.219612 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:00.219583    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8feb759-1de2-4816-93bc-a80445233f5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8118c657-bj8cc\" (UID: \"f8feb759-1de2-4816-93bc-a80445233f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc"
Sep 08 10:22:00.220173 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:00.220156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8feb759-1de2-4816-93bc-a80445233f5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8118c657-bj8cc\" (UID: \"f8feb759-1de2-4816-93bc-a80445233f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc"
Sep 08 10:22:00.390121 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:00.390053    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc"
Sep 08 10:22:00.662068 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:00.662040    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc"]
Sep 08 10:22:00.665762 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:22:00.665731    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8feb759_1de2_4816_93bc_a80445233f5e.slice/crio-deb9f1b03be665af08622cbeb2c595b75326a61bb3b16373ce02ce1e8d3e98c1 WatchSource:0}: Error finding container deb9f1b03be665af08622cbeb2c595b75326a61bb3b16373ce02ce1e8d3e98c1: Status 404 returned error can't find the container with id deb9f1b03be665af08622cbeb2c595b75326a61bb3b16373ce02ce1e8d3e98c1
Sep 08 10:22:00.967551 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:00.967523    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc" event={"ID":"f8feb759-1de2-4816-93bc-a80445233f5e","Type":"ContainerStarted","Data":"a75868019a74463619450d7af8144496e5ae12f992ea00d6b2255e4c03357bdc"}
Sep 08 10:22:00.967649 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:00.967575    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc" event={"ID":"f8feb759-1de2-4816-93bc-a80445233f5e","Type":"ContainerStarted","Data":"deb9f1b03be665af08622cbeb2c595b75326a61bb3b16373ce02ce1e8d3e98c1"}
Sep 08 10:22:00.981600 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:00.981558    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc" podStartSLOduration=0.981544295 podStartE2EDuration="981.544295ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:22:00.980401527 +0000 UTC m=+1240.821625921" watchObservedRunningTime="2026-09-08 10:22:00.981544295 +0000 UTC m=+1240.822768677"
Sep 08 10:22:01.974837 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:01.974805    2559 generic.go:356] "Generic (PLEG): container finished" podID="f8feb759-1de2-4816-93bc-a80445233f5e" containerID="a75868019a74463619450d7af8144496e5ae12f992ea00d6b2255e4c03357bdc" exitCode=0
Sep 08 10:22:01.975215 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:01.974892    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc" event={"ID":"f8feb759-1de2-4816-93bc-a80445233f5e","Type":"ContainerDied","Data":"a75868019a74463619450d7af8144496e5ae12f992ea00d6b2255e4c03357bdc"}
Sep 08 10:22:03.116406 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:03.116381    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc"
Sep 08 10:22:03.148513 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:03.148480    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8feb759-1de2-4816-93bc-a80445233f5e-maas-api-service-ca\") pod \"f8feb759-1de2-4816-93bc-a80445233f5e\" (UID: \"f8feb759-1de2-4816-93bc-a80445233f5e\") "
Sep 08 10:22:03.148797 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:03.148777    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8feb759-1de2-4816-93bc-a80445233f5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8feb759-1de2-4816-93bc-a80445233f5e" (UID: "f8feb759-1de2-4816-93bc-a80445233f5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:22:03.249977 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:03.249913    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8feb759-1de2-4816-93bc-a80445233f5e-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:22:03.987066 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:03.987035    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc"
Sep 08 10:22:03.987330 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:03.987076    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc" event={"ID":"f8feb759-1de2-4816-93bc-a80445233f5e","Type":"ContainerDied","Data":"deb9f1b03be665af08622cbeb2c595b75326a61bb3b16373ce02ce1e8d3e98c1"}
Sep 08 10:22:03.987330 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:03.987122    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deb9f1b03be665af08622cbeb2c595b75326a61bb3b16373ce02ce1e8d3e98c1"
Sep 08 10:22:15.801840 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:15.801642    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"]
Sep 08 10:22:15.802287 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:15.801915    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd" podUID="22fa1f43-aff8-4f59-b6d7-e053a0fc0d22" containerName="payload-pre-processing" containerID="cri-o://acc10ba5490fa1595f698e5e6dd15a2ad237eb14e97bc7df552954e222ac0e50" gracePeriod=30
Sep 08 10:22:16.014474 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:16.014446    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc"]
Sep 08 10:22:16.028879 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:16.028855    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8118c657-bj8cc"]
Sep 08 10:22:16.063150 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:16.063096    2559 generic.go:356] "Generic (PLEG): container finished" podID="22fa1f43-aff8-4f59-b6d7-e053a0fc0d22" containerID="acc10ba5490fa1595f698e5e6dd15a2ad237eb14e97bc7df552954e222ac0e50" exitCode=0
Sep 08 10:22:16.063288 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:16.063169    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd" event={"ID":"22fa1f43-aff8-4f59-b6d7-e053a0fc0d22","Type":"ContainerDied","Data":"acc10ba5490fa1595f698e5e6dd15a2ad237eb14e97bc7df552954e222ac0e50"}
Sep 08 10:22:16.063288 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:16.063225    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd" event={"ID":"22fa1f43-aff8-4f59-b6d7-e053a0fc0d22","Type":"ContainerDied","Data":"dfae41dc98f07da4c725de5dd212db581c72ae8b7012f512c30f9461f6383ae0"}
Sep 08 10:22:16.063288 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:16.063278    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfae41dc98f07da4c725de5dd212db581c72ae8b7012f512c30f9461f6383ae0"
Sep 08 10:22:16.064564 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:16.064547    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"
Sep 08 10:22:16.166062 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:16.166040    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpxnq\" (UniqueName: \"kubernetes.io/projected/22fa1f43-aff8-4f59-b6d7-e053a0fc0d22-kube-api-access-rpxnq\") pod \"22fa1f43-aff8-4f59-b6d7-e053a0fc0d22\" (UID: \"22fa1f43-aff8-4f59-b6d7-e053a0fc0d22\") "
Sep 08 10:22:16.166188 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:16.166115    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fa1f43-aff8-4f59-b6d7-e053a0fc0d22-plugins-config-volume\") pod \"22fa1f43-aff8-4f59-b6d7-e053a0fc0d22\" (UID: \"22fa1f43-aff8-4f59-b6d7-e053a0fc0d22\") "
Sep 08 10:22:16.166502 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:16.166478    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fa1f43-aff8-4f59-b6d7-e053a0fc0d22-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22fa1f43-aff8-4f59-b6d7-e053a0fc0d22" (UID: "22fa1f43-aff8-4f59-b6d7-e053a0fc0d22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:22:16.168078 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:16.168061    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fa1f43-aff8-4f59-b6d7-e053a0fc0d22-kube-api-access-rpxnq" (OuterVolumeSpecName: "kube-api-access-rpxnq") pod "22fa1f43-aff8-4f59-b6d7-e053a0fc0d22" (UID: "22fa1f43-aff8-4f59-b6d7-e053a0fc0d22"). InnerVolumeSpecName "kube-api-access-rpxnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:22:16.267033 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:16.267007    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fa1f43-aff8-4f59-b6d7-e053a0fc0d22-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:22:16.267033 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:16.267029    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpxnq\" (UniqueName: \"kubernetes.io/projected/22fa1f43-aff8-4f59-b6d7-e053a0fc0d22-kube-api-access-rpxnq\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:22:16.637253 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:16.637217    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8feb759-1de2-4816-93bc-a80445233f5e" path="/var/lib/kubelet/pods/f8feb759-1de2-4816-93bc-a80445233f5e/volumes"
Sep 08 10:22:17.068796 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:17.068761    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"
Sep 08 10:22:17.104092 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:17.104064    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"]
Sep 08 10:22:17.115200 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:17.115177    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8118c657-6c87877b94-z6btd"]
Sep 08 10:22:18.636835 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:18.636806    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22fa1f43-aff8-4f59-b6d7-e053a0fc0d22" path="/var/lib/kubelet/pods/22fa1f43-aff8-4f59-b6d7-e053a0fc0d22/volumes"
Sep 08 10:22:20.831348 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:20.831319    2559 scope.go:117] "RemoveContainer" containerID="c976a505884d76428bd0c875f6a4c002f4157b12ad7ff205fb81c0d7c75691b6"
Sep 08 10:22:20.840144 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:20.840126    2559 scope.go:117] "RemoveContainer" containerID="cf6c2efb1d26af46cddf5b7276104ef726239b00da7dc242e4544f173fcf7613"
Sep 08 10:22:24.525151 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:24.525116    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"]
Sep 08 10:22:39.545394 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.545327    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb" podUID="96a7b577-5e8d-410d-98b9-08118481123b" containerName="main" containerID="cri-o://4891b59ca498c33ac69f395035c83c67e2f6ef156e9e4583e8b127606a2953c3" gracePeriod=45
Sep 08 10:22:39.793450 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.793421    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:22:39.879885 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.879817    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-model-cache\") pod \"96a7b577-5e8d-410d-98b9-08118481123b\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") "
Sep 08 10:22:39.880008 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.879916    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a7b577-5e8d-410d-98b9-08118481123b-tls-certs\") pod \"96a7b577-5e8d-410d-98b9-08118481123b\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") "
Sep 08 10:22:39.880008 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.879967    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-home\") pod \"96a7b577-5e8d-410d-98b9-08118481123b\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") "
Sep 08 10:22:39.880008 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.879998    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjjrf\" (UniqueName: \"kubernetes.io/projected/96a7b577-5e8d-410d-98b9-08118481123b-kube-api-access-cjjrf\") pod \"96a7b577-5e8d-410d-98b9-08118481123b\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") "
Sep 08 10:22:39.880141 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.880042    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-dshm\") pod \"96a7b577-5e8d-410d-98b9-08118481123b\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") "
Sep 08 10:22:39.880141 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.880087    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-tmp-dir\") pod \"96a7b577-5e8d-410d-98b9-08118481123b\" (UID: \"96a7b577-5e8d-410d-98b9-08118481123b\") "
Sep 08 10:22:39.880141 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.880089    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-model-cache" (OuterVolumeSpecName: "model-cache") pod "96a7b577-5e8d-410d-98b9-08118481123b" (UID: "96a7b577-5e8d-410d-98b9-08118481123b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:22:39.880341 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.880288    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-home" (OuterVolumeSpecName: "home") pod "96a7b577-5e8d-410d-98b9-08118481123b" (UID: "96a7b577-5e8d-410d-98b9-08118481123b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:22:39.880425 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.880398    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96a7b577-5e8d-410d-98b9-08118481123b" (UID: "96a7b577-5e8d-410d-98b9-08118481123b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:22:39.880513 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.880495    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-home\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:22:39.880561 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.880520    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-tmp-dir\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:22:39.880561 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.880535    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-model-cache\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:22:39.882167 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.882142    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a7b577-5e8d-410d-98b9-08118481123b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96a7b577-5e8d-410d-98b9-08118481123b" (UID: "96a7b577-5e8d-410d-98b9-08118481123b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:22:39.882273 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.882145    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-dshm" (OuterVolumeSpecName: "dshm") pod "96a7b577-5e8d-410d-98b9-08118481123b" (UID: "96a7b577-5e8d-410d-98b9-08118481123b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:22:39.882273 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.882194    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a7b577-5e8d-410d-98b9-08118481123b-kube-api-access-cjjrf" (OuterVolumeSpecName: "kube-api-access-cjjrf") pod "96a7b577-5e8d-410d-98b9-08118481123b" (UID: "96a7b577-5e8d-410d-98b9-08118481123b"). InnerVolumeSpecName "kube-api-access-cjjrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:22:39.981959 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.981934    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjjrf\" (UniqueName: \"kubernetes.io/projected/96a7b577-5e8d-410d-98b9-08118481123b-kube-api-access-cjjrf\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:22:39.981959 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.981956    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a7b577-5e8d-410d-98b9-08118481123b-dshm\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:22:39.982099 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:39.981966    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a7b577-5e8d-410d-98b9-08118481123b-tls-certs\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:22:40.203661 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:40.203633    2559 generic.go:356] "Generic (PLEG): container finished" podID="96a7b577-5e8d-410d-98b9-08118481123b" containerID="4891b59ca498c33ac69f395035c83c67e2f6ef156e9e4583e8b127606a2953c3" exitCode=0
Sep 08 10:22:40.203786 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:40.203678    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"
Sep 08 10:22:40.203786 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:40.203717    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb" event={"ID":"96a7b577-5e8d-410d-98b9-08118481123b","Type":"ContainerDied","Data":"4891b59ca498c33ac69f395035c83c67e2f6ef156e9e4583e8b127606a2953c3"}
Sep 08 10:22:40.203786 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:40.203759    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb" event={"ID":"96a7b577-5e8d-410d-98b9-08118481123b","Type":"ContainerDied","Data":"c6eb34035ce9bfb98d1da8de4cfb0b25c828e6f90a6f4a5ee4e6089e7d1567e2"}
Sep 08 10:22:40.203786 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:40.203777    2559 scope.go:117] "RemoveContainer" containerID="4891b59ca498c33ac69f395035c83c67e2f6ef156e9e4583e8b127606a2953c3"
Sep 08 10:22:40.212966 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:40.212951    2559 scope.go:117] "RemoveContainer" containerID="4891b59ca498c33ac69f395035c83c67e2f6ef156e9e4583e8b127606a2953c3"
Sep 08 10:22:40.213220 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:22:40.213202    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4891b59ca498c33ac69f395035c83c67e2f6ef156e9e4583e8b127606a2953c3\": container with ID starting with 4891b59ca498c33ac69f395035c83c67e2f6ef156e9e4583e8b127606a2953c3 not found: ID does not exist" containerID="4891b59ca498c33ac69f395035c83c67e2f6ef156e9e4583e8b127606a2953c3"
Sep 08 10:22:40.213306 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:40.213229    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4891b59ca498c33ac69f395035c83c67e2f6ef156e9e4583e8b127606a2953c3"} err="failed to get container status \"4891b59ca498c33ac69f395035c83c67e2f6ef156e9e4583e8b127606a2953c3\": rpc error: code = NotFound desc = could not find container \"4891b59ca498c33ac69f395035c83c67e2f6ef156e9e4583e8b127606a2953c3\": container with ID starting with 4891b59ca498c33ac69f395035c83c67e2f6ef156e9e4583e8b127606a2953c3 not found: ID does not exist"
Sep 08 10:22:40.224964 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:40.224944    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"]
Sep 08 10:22:40.226857 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:40.226839    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8118c657/e2e-lifecycle-model-8118c657-kserve-84b67f9ccf-r4wtb"]
Sep 08 10:22:40.637225 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:22:40.637152    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96a7b577-5e8d-410d-98b9-08118481123b" path="/var/lib/kubelet/pods/96a7b577-5e8d-410d-98b9-08118481123b/volumes"
Sep 08 10:23:20.864349 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:20.864319    2559 scope.go:117] "RemoveContainer" containerID="297bc6390e3d5adcb064f7cb2401e94fe2dd44beb42f4f9c8a156c52355dd68c"
Sep 08 10:23:30.830452 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:30.830044    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"]
Sep 08 10:23:31.944685 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:31.944647    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"]
Sep 08 10:23:45.850410 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:45.850310    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m" podUID="8e8158d3-d44b-4bdc-aae6-0125b02b69fb" containerName="main" containerID="cri-o://00539c6a95e680e3b9f961fd8915c6937c6a16f80b377531e47335bbe03e7cbd" gracePeriod=45
Sep 08 10:23:46.102351 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.102294    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:23:46.143285 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.143262    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-home\") pod \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") "
Sep 08 10:23:46.143419 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.143342    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q62mv\" (UniqueName: \"kubernetes.io/projected/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-kube-api-access-q62mv\") pod \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") "
Sep 08 10:23:46.143419 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.143379    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-dshm\") pod \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") "
Sep 08 10:23:46.143535 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.143441    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-home" (OuterVolumeSpecName: "home") pod "8e8158d3-d44b-4bdc-aae6-0125b02b69fb" (UID: "8e8158d3-d44b-4bdc-aae6-0125b02b69fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:23:46.143535 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.143454    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-model-cache\") pod \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") "
Sep 08 10:23:46.143535 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.143529    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-tls-certs\") pod \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") "
Sep 08 10:23:46.143694 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.143570    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-tmp-dir\") pod \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\" (UID: \"8e8158d3-d44b-4bdc-aae6-0125b02b69fb\") "
Sep 08 10:23:46.143694 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.143678    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e8158d3-d44b-4bdc-aae6-0125b02b69fb" (UID: "8e8158d3-d44b-4bdc-aae6-0125b02b69fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:23:46.143876 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.143835    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e8158d3-d44b-4bdc-aae6-0125b02b69fb" (UID: "8e8158d3-d44b-4bdc-aae6-0125b02b69fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:23:46.143993 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.143959    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-model-cache\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:23:46.143993 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.143977    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-tmp-dir\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:23:46.143993 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.143990    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-home\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:23:46.145697 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.145675    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e8158d3-d44b-4bdc-aae6-0125b02b69fb" (UID: "8e8158d3-d44b-4bdc-aae6-0125b02b69fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:23:46.145860 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.145843    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-dshm" (OuterVolumeSpecName: "dshm") pod "8e8158d3-d44b-4bdc-aae6-0125b02b69fb" (UID: "8e8158d3-d44b-4bdc-aae6-0125b02b69fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:23:46.145924 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.145855    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-kube-api-access-q62mv" (OuterVolumeSpecName: "kube-api-access-q62mv") pod "8e8158d3-d44b-4bdc-aae6-0125b02b69fb" (UID: "8e8158d3-d44b-4bdc-aae6-0125b02b69fb"). InnerVolumeSpecName "kube-api-access-q62mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:23:46.244634 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.244609    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-dshm\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:23:46.244634 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.244632    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-tls-certs\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:23:46.244805 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.244642    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q62mv\" (UniqueName: \"kubernetes.io/projected/8e8158d3-d44b-4bdc-aae6-0125b02b69fb-kube-api-access-q62mv\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:23:46.561826 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.561790    2559 generic.go:356] "Generic (PLEG): container finished" podID="8e8158d3-d44b-4bdc-aae6-0125b02b69fb" containerID="00539c6a95e680e3b9f961fd8915c6937c6a16f80b377531e47335bbe03e7cbd" exitCode=0
Sep 08 10:23:46.562020 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.561835    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"
Sep 08 10:23:46.562020 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.561874    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m" event={"ID":"8e8158d3-d44b-4bdc-aae6-0125b02b69fb","Type":"ContainerDied","Data":"00539c6a95e680e3b9f961fd8915c6937c6a16f80b377531e47335bbe03e7cbd"}
Sep 08 10:23:46.562020 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.561917    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m" event={"ID":"8e8158d3-d44b-4bdc-aae6-0125b02b69fb","Type":"ContainerDied","Data":"4980c63213345f1e20ed8241c9c27a2d83ea3fc7185adc888f7d71a9d2bdee8e"}
Sep 08 10:23:46.562020 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.561934    2559 scope.go:117] "RemoveContainer" containerID="00539c6a95e680e3b9f961fd8915c6937c6a16f80b377531e47335bbe03e7cbd"
Sep 08 10:23:46.571221 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.571207    2559 scope.go:117] "RemoveContainer" containerID="00539c6a95e680e3b9f961fd8915c6937c6a16f80b377531e47335bbe03e7cbd"
Sep 08 10:23:46.571501 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:23:46.571483    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00539c6a95e680e3b9f961fd8915c6937c6a16f80b377531e47335bbe03e7cbd\": container with ID starting with 00539c6a95e680e3b9f961fd8915c6937c6a16f80b377531e47335bbe03e7cbd not found: ID does not exist" containerID="00539c6a95e680e3b9f961fd8915c6937c6a16f80b377531e47335bbe03e7cbd"
Sep 08 10:23:46.571543 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.571508    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00539c6a95e680e3b9f961fd8915c6937c6a16f80b377531e47335bbe03e7cbd"} err="failed to get container status \"00539c6a95e680e3b9f961fd8915c6937c6a16f80b377531e47335bbe03e7cbd\": rpc error: code = NotFound desc = could not find container \"00539c6a95e680e3b9f961fd8915c6937c6a16f80b377531e47335bbe03e7cbd\": container with ID starting with 00539c6a95e680e3b9f961fd8915c6937c6a16f80b377531e47335bbe03e7cbd not found: ID does not exist"
Sep 08 10:23:46.583358 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.583337    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"]
Sep 08 10:23:46.587093 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.587075    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/auth-test-model-71a7a1-kserve-78d4dcf587-4tm9m"]
Sep 08 10:23:46.636625 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:46.636598    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e8158d3-d44b-4bdc-aae6-0125b02b69fb" path="/var/lib/kubelet/pods/8e8158d3-d44b-4bdc-aae6-0125b02b69fb/volumes"
Sep 08 10:23:47.507705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.507668    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn" podUID="fd96c96c-8870-4020-9b8b-06f15575c819" containerName="main" containerID="cri-o://ee6b3fe59974131cf3ba968db291d2124afcae9fa10f24f5d76d98a6b8ccdc70" gracePeriod=45
Sep 08 10:23:47.784096 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.784061    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:23:47.859448 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.859419    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-tmp-dir\") pod \"fd96c96c-8870-4020-9b8b-06f15575c819\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") "
Sep 08 10:23:47.859607 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.859510    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnz7w\" (UniqueName: \"kubernetes.io/projected/fd96c96c-8870-4020-9b8b-06f15575c819-kube-api-access-dnz7w\") pod \"fd96c96c-8870-4020-9b8b-06f15575c819\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") "
Sep 08 10:23:47.859607 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.859542    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd96c96c-8870-4020-9b8b-06f15575c819-tls-certs\") pod \"fd96c96c-8870-4020-9b8b-06f15575c819\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") "
Sep 08 10:23:47.859607 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.859582    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-home\") pod \"fd96c96c-8870-4020-9b8b-06f15575c819\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") "
Sep 08 10:23:47.859782 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.859686    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd96c96c-8870-4020-9b8b-06f15575c819" (UID: "fd96c96c-8870-4020-9b8b-06f15575c819"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:23:47.859782 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.859705    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-dshm\") pod \"fd96c96c-8870-4020-9b8b-06f15575c819\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") "
Sep 08 10:23:47.859782 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.859741    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-model-cache\") pod \"fd96c96c-8870-4020-9b8b-06f15575c819\" (UID: \"fd96c96c-8870-4020-9b8b-06f15575c819\") "
Sep 08 10:23:47.859943 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.859846    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-home" (OuterVolumeSpecName: "home") pod "fd96c96c-8870-4020-9b8b-06f15575c819" (UID: "fd96c96c-8870-4020-9b8b-06f15575c819"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:23:47.860061 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.860046    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-tmp-dir\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:23:47.860119 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.860062    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-home\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:23:47.860119 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.860064    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd96c96c-8870-4020-9b8b-06f15575c819" (UID: "fd96c96c-8870-4020-9b8b-06f15575c819"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:23:47.861851 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.861828    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-dshm" (OuterVolumeSpecName: "dshm") pod "fd96c96c-8870-4020-9b8b-06f15575c819" (UID: "fd96c96c-8870-4020-9b8b-06f15575c819"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:23:47.861947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.861889    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd96c96c-8870-4020-9b8b-06f15575c819-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd96c96c-8870-4020-9b8b-06f15575c819" (UID: "fd96c96c-8870-4020-9b8b-06f15575c819"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:23:47.861947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.861900    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd96c96c-8870-4020-9b8b-06f15575c819-kube-api-access-dnz7w" (OuterVolumeSpecName: "kube-api-access-dnz7w") pod "fd96c96c-8870-4020-9b8b-06f15575c819" (UID: "fd96c96c-8870-4020-9b8b-06f15575c819"). InnerVolumeSpecName "kube-api-access-dnz7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:23:47.961511 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.961487    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnz7w\" (UniqueName: \"kubernetes.io/projected/fd96c96c-8870-4020-9b8b-06f15575c819-kube-api-access-dnz7w\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:23:47.961511 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.961511    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd96c96c-8870-4020-9b8b-06f15575c819-tls-certs\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:23:47.961668 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.961520    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-dshm\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:23:47.961668 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:47.961528    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd96c96c-8870-4020-9b8b-06f15575c819-model-cache\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:23:48.574511 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:48.574472    2559 generic.go:356] "Generic (PLEG): container finished" podID="fd96c96c-8870-4020-9b8b-06f15575c819" containerID="ee6b3fe59974131cf3ba968db291d2124afcae9fa10f24f5d76d98a6b8ccdc70" exitCode=0
Sep 08 10:23:48.574977 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:48.574541    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"
Sep 08 10:23:48.574977 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:48.574550    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn" event={"ID":"fd96c96c-8870-4020-9b8b-06f15575c819","Type":"ContainerDied","Data":"ee6b3fe59974131cf3ba968db291d2124afcae9fa10f24f5d76d98a6b8ccdc70"}
Sep 08 10:23:48.574977 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:48.574587    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn" event={"ID":"fd96c96c-8870-4020-9b8b-06f15575c819","Type":"ContainerDied","Data":"de1abc0c1f587420d26f2414f677eacec504daa64ae3a5ac491f4cdd7130bcbb"}
Sep 08 10:23:48.574977 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:48.574605    2559 scope.go:117] "RemoveContainer" containerID="ee6b3fe59974131cf3ba968db291d2124afcae9fa10f24f5d76d98a6b8ccdc70"
Sep 08 10:23:48.584321 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:48.584306    2559 scope.go:117] "RemoveContainer" containerID="ee6b3fe59974131cf3ba968db291d2124afcae9fa10f24f5d76d98a6b8ccdc70"
Sep 08 10:23:48.584573 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:23:48.584557    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6b3fe59974131cf3ba968db291d2124afcae9fa10f24f5d76d98a6b8ccdc70\": container with ID starting with ee6b3fe59974131cf3ba968db291d2124afcae9fa10f24f5d76d98a6b8ccdc70 not found: ID does not exist" containerID="ee6b3fe59974131cf3ba968db291d2124afcae9fa10f24f5d76d98a6b8ccdc70"
Sep 08 10:23:48.584672 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:48.584578    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6b3fe59974131cf3ba968db291d2124afcae9fa10f24f5d76d98a6b8ccdc70"} err="failed to get container status \"ee6b3fe59974131cf3ba968db291d2124afcae9fa10f24f5d76d98a6b8ccdc70\": rpc error: code = NotFound desc = could not find container \"ee6b3fe59974131cf3ba968db291d2124afcae9fa10f24f5d76d98a6b8ccdc70\": container with ID starting with ee6b3fe59974131cf3ba968db291d2124afcae9fa10f24f5d76d98a6b8ccdc70 not found: ID does not exist"
Sep 08 10:23:48.600727 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:48.600692    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"]
Sep 08 10:23:48.602881 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:48.602864    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d6e53c/auth-test-model-d6e53c-kserve-5bdf858747-9tngn"]
Sep 08 10:23:48.636387 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:23:48.636363    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd96c96c-8870-4020-9b8b-06f15575c819" path="/var/lib/kubelet/pods/fd96c96c-8870-4020-9b8b-06f15575c819/volumes"
Sep 08 10:24:04.637756 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.637727    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"]
Sep 08 10:24:04.638182 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638102    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd96c96c-8870-4020-9b8b-06f15575c819" containerName="main"
Sep 08 10:24:04.638182 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638112    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd96c96c-8870-4020-9b8b-06f15575c819" containerName="main"
Sep 08 10:24:04.638182 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638125    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8feb759-1de2-4816-93bc-a80445233f5e" containerName="revoke-keys"
Sep 08 10:24:04.638182 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638130    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8feb759-1de2-4816-93bc-a80445233f5e" containerName="revoke-keys"
Sep 08 10:24:04.638182 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638141    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a7b577-5e8d-410d-98b9-08118481123b" containerName="main"
Sep 08 10:24:04.638182 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638146    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a7b577-5e8d-410d-98b9-08118481123b" containerName="main"
Sep 08 10:24:04.638182 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638153    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22fa1f43-aff8-4f59-b6d7-e053a0fc0d22" containerName="payload-pre-processing"
Sep 08 10:24:04.638182 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638158    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="22fa1f43-aff8-4f59-b6d7-e053a0fc0d22" containerName="payload-pre-processing"
Sep 08 10:24:04.638182 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638168    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e8158d3-d44b-4bdc-aae6-0125b02b69fb" containerName="main"
Sep 08 10:24:04.638182 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638173    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e8158d3-d44b-4bdc-aae6-0125b02b69fb" containerName="main"
Sep 08 10:24:04.638508 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638257    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="22fa1f43-aff8-4f59-b6d7-e053a0fc0d22" containerName="payload-pre-processing"
Sep 08 10:24:04.638508 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638267    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a7b577-5e8d-410d-98b9-08118481123b" containerName="main"
Sep 08 10:24:04.638508 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638273    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8feb759-1de2-4816-93bc-a80445233f5e" containerName="revoke-keys"
Sep 08 10:24:04.638508 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638283    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e8158d3-d44b-4bdc-aae6-0125b02b69fb" containerName="main"
Sep 08 10:24:04.638508 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.638288    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd96c96c-8870-4020-9b8b-06f15575c819" containerName="main"
Sep 08 10:24:04.641411 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.641395    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.643908 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.643877    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-71a7a1\"/\"kube-root-ca.crt\""
Sep 08 10:24:04.644942 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.644920    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-71a7a1\"/\"openshift-service-ca.crt\""
Sep 08 10:24:04.645164 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.645140    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-71a7a1\"/\"default-dockercfg-rf8zh\""
Sep 08 10:24:04.645465 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.645441    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-71a7a1\"/\"sub-test-model-71a7a1-kserve-self-signed-certs\""
Sep 08 10:24:04.647919 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.647901    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"]
Sep 08 10:24:04.800475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.800442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbgd4\" (UniqueName: \"kubernetes.io/projected/01c2011f-4ab1-471d-86b1-40bffa377bb1-kube-api-access-cbgd4\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.800475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.800477    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-model-cache\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.800656 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.800495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-dshm\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.800656 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.800563    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-home\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.800656 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.800606    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c2011f-4ab1-471d-86b1-40bffa377bb1-tls-certs\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.800656 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.800637    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-tmp-dir\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.901497 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.901423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c2011f-4ab1-471d-86b1-40bffa377bb1-tls-certs\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.901497 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.901468    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-tmp-dir\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.901719 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.901525    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbgd4\" (UniqueName: \"kubernetes.io/projected/01c2011f-4ab1-471d-86b1-40bffa377bb1-kube-api-access-cbgd4\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.901719 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.901556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-model-cache\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.901719 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.901583    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-dshm\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.901719 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.901626    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-home\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.901991 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.901954    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-tmp-dir\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.901991 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.901982    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-model-cache\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.902158 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.902005    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-home\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.903788 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.903769    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-dshm\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.903953 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.903937    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c2011f-4ab1-471d-86b1-40bffa377bb1-tls-certs\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.909160 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.909140    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbgd4\" (UniqueName: \"kubernetes.io/projected/01c2011f-4ab1-471d-86b1-40bffa377bb1-kube-api-access-cbgd4\") pod \"sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:04.951990 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:04.951967    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:05.227922 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:05.227884    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"]
Sep 08 10:24:05.229271 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:24:05.229218    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01c2011f_4ab1_471d_86b1_40bffa377bb1.slice/crio-c17ba05da3a94ebba4fcabcf2760d4aa3766e089d405932db6a0260f83129bed WatchSource:0}: Error finding container c17ba05da3a94ebba4fcabcf2760d4aa3766e089d405932db6a0260f83129bed: Status 404 returned error can't find the container with id c17ba05da3a94ebba4fcabcf2760d4aa3766e089d405932db6a0260f83129bed
Sep 08 10:24:05.668279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:05.668223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf" event={"ID":"01c2011f-4ab1-471d-86b1-40bffa377bb1","Type":"ContainerStarted","Data":"e04afa955a2cfe25f3b4aa70534f082743dab37d037df54f56ea231ccb452743"}
Sep 08 10:24:05.668279 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:05.668278    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf" event={"ID":"01c2011f-4ab1-471d-86b1-40bffa377bb1","Type":"ContainerStarted","Data":"c17ba05da3a94ebba4fcabcf2760d4aa3766e089d405932db6a0260f83129bed"}
Sep 08 10:24:05.685577 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:05.685533    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf" podStartSLOduration=1.685519274 podStartE2EDuration="1.685519274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:24:05.68415469 +0000 UTC m=+1365.525379074" watchObservedRunningTime="2026-09-08 10:24:05.685519274 +0000 UTC m=+1365.526743657"
Sep 08 10:24:08.280653 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:08.280610    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"]
Sep 08 10:24:08.285646 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:08.285618    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"
Sep 08 10:24:08.288152 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:08.288124    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-22af43\""
Sep 08 10:24:08.288345 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:08.288237    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-22af43-dockercfg-42vhz\""
Sep 08 10:24:08.296447 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:08.296424    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"]
Sep 08 10:24:08.433091 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:08.433053    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgr77\" (UniqueName: \"kubernetes.io/projected/ff7ec9fa-b726-42d3-979c-2ba885817bfa-kube-api-access-cgr77\") pod \"payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2\" (UID: \"ff7ec9fa-b726-42d3-979c-2ba885817bfa\") " pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"
Sep 08 10:24:08.433283 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:08.433167    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7ec9fa-b726-42d3-979c-2ba885817bfa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2\" (UID: \"ff7ec9fa-b726-42d3-979c-2ba885817bfa\") " pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"
Sep 08 10:24:08.534672 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:08.534596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgr77\" (UniqueName: \"kubernetes.io/projected/ff7ec9fa-b726-42d3-979c-2ba885817bfa-kube-api-access-cgr77\") pod \"payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2\" (UID: \"ff7ec9fa-b726-42d3-979c-2ba885817bfa\") " pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"
Sep 08 10:24:08.534811 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:08.534686    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7ec9fa-b726-42d3-979c-2ba885817bfa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2\" (UID: \"ff7ec9fa-b726-42d3-979c-2ba885817bfa\") " pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"
Sep 08 10:24:08.535413 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:08.535394    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7ec9fa-b726-42d3-979c-2ba885817bfa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2\" (UID: \"ff7ec9fa-b726-42d3-979c-2ba885817bfa\") " pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"
Sep 08 10:24:08.542490 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:08.542471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgr77\" (UniqueName: \"kubernetes.io/projected/ff7ec9fa-b726-42d3-979c-2ba885817bfa-kube-api-access-cgr77\") pod \"payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2\" (UID: \"ff7ec9fa-b726-42d3-979c-2ba885817bfa\") " pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"
Sep 08 10:24:08.598574 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:08.598550    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"
Sep 08 10:24:08.872561 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:08.872540    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"]
Sep 08 10:24:08.874384 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:24:08.874358    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7ec9fa_b726_42d3_979c_2ba885817bfa.slice/crio-9c5849d807117b3e7416cc3978e42d85125c85d9d9c4fb2129df4e775399f684 WatchSource:0}: Error finding container 9c5849d807117b3e7416cc3978e42d85125c85d9d9c4fb2129df4e775399f684: Status 404 returned error can't find the container with id 9c5849d807117b3e7416cc3978e42d85125c85d9d9c4fb2129df4e775399f684
Sep 08 10:24:09.694390 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:09.694348    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2" event={"ID":"ff7ec9fa-b726-42d3-979c-2ba885817bfa","Type":"ContainerStarted","Data":"ad47fa01a1aa4141bdf0978f5d556fbdbd09033cebd0b392086c7fd810a44b87"}
Sep 08 10:24:09.694839 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:09.694418    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"
Sep 08 10:24:09.694839 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:09.694441    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2" event={"ID":"ff7ec9fa-b726-42d3-979c-2ba885817bfa","Type":"ContainerStarted","Data":"9c5849d807117b3e7416cc3978e42d85125c85d9d9c4fb2129df4e775399f684"}
Sep 08 10:24:09.712275 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:09.712208    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2" podStartSLOduration=1.712193494 podStartE2EDuration="1.712193494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:24:09.710104664 +0000 UTC m=+1369.551329071" watchObservedRunningTime="2026-09-08 10:24:09.712193494 +0000 UTC m=+1369.553417876"
Sep 08 10:24:14.953015 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:14.952984    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:14.953490 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:14.953032    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:14.965434 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:14.965408    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:15.740506 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:15.740473    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:24:20.700965 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:20.700935    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"
Sep 08 10:24:20.889598 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:20.889567    2559 scope.go:117] "RemoveContainer" containerID="d0f657bb6a047c0747c9eb6ecdcf9536377b6820f629c3015281ef1598628b5e"
Sep 08 10:24:37.394261 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:37.394206    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"]
Sep 08 10:24:37.396985 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:37.396969    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"
Sep 08 10:24:37.399634 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:37.399612    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9e30f6\""
Sep 08 10:24:37.399756 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:37.399665    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9e30f6-dockercfg-2bk6k\""
Sep 08 10:24:37.412021 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:37.412000    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"]
Sep 08 10:24:37.490919 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:37.490890    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27ddf73-94dd-4424-a19b-2c6889b41359-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp\" (UID: \"b27ddf73-94dd-4424-a19b-2c6889b41359\") " pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"
Sep 08 10:24:37.491069 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:37.490923    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54g7z\" (UniqueName: \"kubernetes.io/projected/b27ddf73-94dd-4424-a19b-2c6889b41359-kube-api-access-54g7z\") pod \"payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp\" (UID: \"b27ddf73-94dd-4424-a19b-2c6889b41359\") " pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"
Sep 08 10:24:37.591330 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:37.591292    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27ddf73-94dd-4424-a19b-2c6889b41359-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp\" (UID: \"b27ddf73-94dd-4424-a19b-2c6889b41359\") " pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"
Sep 08 10:24:37.591330 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:37.591337    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54g7z\" (UniqueName: \"kubernetes.io/projected/b27ddf73-94dd-4424-a19b-2c6889b41359-kube-api-access-54g7z\") pod \"payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp\" (UID: \"b27ddf73-94dd-4424-a19b-2c6889b41359\") " pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"
Sep 08 10:24:37.591959 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:37.591939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27ddf73-94dd-4424-a19b-2c6889b41359-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp\" (UID: \"b27ddf73-94dd-4424-a19b-2c6889b41359\") " pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"
Sep 08 10:24:37.600289 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:37.600239    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54g7z\" (UniqueName: \"kubernetes.io/projected/b27ddf73-94dd-4424-a19b-2c6889b41359-kube-api-access-54g7z\") pod \"payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp\" (UID: \"b27ddf73-94dd-4424-a19b-2c6889b41359\") " pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"
Sep 08 10:24:37.707550 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:37.707516    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"
Sep 08 10:24:37.979884 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:37.979738    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"]
Sep 08 10:24:37.982735 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:24:37.982706    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27ddf73_94dd_4424_a19b_2c6889b41359.slice/crio-07fda58c252ea80a14ec9b23edf3aca177af876779bf0801d90b3d54e49e7c68 WatchSource:0}: Error finding container 07fda58c252ea80a14ec9b23edf3aca177af876779bf0801d90b3d54e49e7c68: Status 404 returned error can't find the container with id 07fda58c252ea80a14ec9b23edf3aca177af876779bf0801d90b3d54e49e7c68
Sep 08 10:24:38.858928 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:38.858886    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp" event={"ID":"b27ddf73-94dd-4424-a19b-2c6889b41359","Type":"ContainerStarted","Data":"cacd494bd8dc10cd1051e125c53d1410a15208cda6a1827c5d26ef8dffa0abbf"}
Sep 08 10:24:38.859322 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:38.858947    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp" event={"ID":"b27ddf73-94dd-4424-a19b-2c6889b41359","Type":"ContainerStarted","Data":"07fda58c252ea80a14ec9b23edf3aca177af876779bf0801d90b3d54e49e7c68"}
Sep 08 10:24:38.859322 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:38.858987    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"
Sep 08 10:24:38.877218 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:38.877170    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp" podStartSLOduration=1.877155025 podStartE2EDuration="1.877155025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:24:38.875886859 +0000 UTC m=+1398.717111271" watchObservedRunningTime="2026-09-08 10:24:38.877155025 +0000 UTC m=+1398.718379407"
Sep 08 10:24:49.866973 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:24:49.866943    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"
Sep 08 10:25:03.529360 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:03.529324    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52"]
Sep 08 10:25:03.532895 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:03.532868    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52"
Sep 08 10:25:03.535592 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:03.535565    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:25:03.535714 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:03.535591    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:25:03.541896 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:03.541873    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52"]
Sep 08 10:25:03.628063 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:03.628034    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3942bc46-55c0-4b5a-b0c6-9c4880709b7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-22af43-wkx52\" (UID: \"3942bc46-55c0-4b5a-b0c6-9c4880709b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52"
Sep 08 10:25:03.729153 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:03.729127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3942bc46-55c0-4b5a-b0c6-9c4880709b7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-22af43-wkx52\" (UID: \"3942bc46-55c0-4b5a-b0c6-9c4880709b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52"
Sep 08 10:25:03.729742 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:03.729721    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3942bc46-55c0-4b5a-b0c6-9c4880709b7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-22af43-wkx52\" (UID: \"3942bc46-55c0-4b5a-b0c6-9c4880709b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52"
Sep 08 10:25:03.844983 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:03.844929    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52"
Sep 08 10:25:04.116955 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:04.116909    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52"]
Sep 08 10:25:04.119461 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:25:04.119433    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3942bc46_55c0_4b5a_b0c6_9c4880709b7e.slice/crio-1370f3d80af116002bd0399f24d037a33d4fab89cd375af29ad1f31c06021882 WatchSource:0}: Error finding container 1370f3d80af116002bd0399f24d037a33d4fab89cd375af29ad1f31c06021882: Status 404 returned error can't find the container with id 1370f3d80af116002bd0399f24d037a33d4fab89cd375af29ad1f31c06021882
Sep 08 10:25:05.006486 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:05.006453    2559 generic.go:356] "Generic (PLEG): container finished" podID="3942bc46-55c0-4b5a-b0c6-9c4880709b7e" containerID="f8b06be308b4993ec2f8512cc6b186ff73f043993731dddd73a09d7053377ba2" exitCode=0
Sep 08 10:25:05.006883 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:05.006534    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52" event={"ID":"3942bc46-55c0-4b5a-b0c6-9c4880709b7e","Type":"ContainerDied","Data":"f8b06be308b4993ec2f8512cc6b186ff73f043993731dddd73a09d7053377ba2"}
Sep 08 10:25:05.006883 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:05.006586    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52" event={"ID":"3942bc46-55c0-4b5a-b0c6-9c4880709b7e","Type":"ContainerStarted","Data":"1370f3d80af116002bd0399f24d037a33d4fab89cd375af29ad1f31c06021882"}
Sep 08 10:25:06.145155 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:06.145131    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52"
Sep 08 10:25:06.249799 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:06.249772    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3942bc46-55c0-4b5a-b0c6-9c4880709b7e-maas-api-service-ca\") pod \"3942bc46-55c0-4b5a-b0c6-9c4880709b7e\" (UID: \"3942bc46-55c0-4b5a-b0c6-9c4880709b7e\") "
Sep 08 10:25:06.250124 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:06.250103    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3942bc46-55c0-4b5a-b0c6-9c4880709b7e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3942bc46-55c0-4b5a-b0c6-9c4880709b7e" (UID: "3942bc46-55c0-4b5a-b0c6-9c4880709b7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:25:06.351471 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:06.351406    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3942bc46-55c0-4b5a-b0c6-9c4880709b7e-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:25:07.018220 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:07.018188    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52" event={"ID":"3942bc46-55c0-4b5a-b0c6-9c4880709b7e","Type":"ContainerDied","Data":"1370f3d80af116002bd0399f24d037a33d4fab89cd375af29ad1f31c06021882"}
Sep 08 10:25:07.018401 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:07.018222    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52"
Sep 08 10:25:07.018401 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:07.018280    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1370f3d80af116002bd0399f24d037a33d4fab89cd375af29ad1f31c06021882"
Sep 08 10:25:19.420879 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:19.420788    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"]
Sep 08 10:25:19.421361 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:19.421050    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2" podUID="ff7ec9fa-b726-42d3-979c-2ba885817bfa" containerName="payload-processing" containerID="cri-o://ad47fa01a1aa4141bdf0978f5d556fbdbd09033cebd0b392086c7fd810a44b87" gracePeriod=30
Sep 08 10:25:19.604161 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:19.604131    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52"]
Sep 08 10:25:19.607642 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:19.607612    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22af43-wkx52"]
Sep 08 10:25:19.669205 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:19.669182    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"
Sep 08 10:25:19.766578 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:19.766505    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgr77\" (UniqueName: \"kubernetes.io/projected/ff7ec9fa-b726-42d3-979c-2ba885817bfa-kube-api-access-cgr77\") pod \"ff7ec9fa-b726-42d3-979c-2ba885817bfa\" (UID: \"ff7ec9fa-b726-42d3-979c-2ba885817bfa\") "
Sep 08 10:25:19.766578 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:19.766564    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7ec9fa-b726-42d3-979c-2ba885817bfa-plugins-config-volume\") pod \"ff7ec9fa-b726-42d3-979c-2ba885817bfa\" (UID: \"ff7ec9fa-b726-42d3-979c-2ba885817bfa\") "
Sep 08 10:25:19.766969 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:19.766945    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff7ec9fa-b726-42d3-979c-2ba885817bfa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff7ec9fa-b726-42d3-979c-2ba885817bfa" (UID: "ff7ec9fa-b726-42d3-979c-2ba885817bfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:25:19.768421 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:19.768404    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7ec9fa-b726-42d3-979c-2ba885817bfa-kube-api-access-cgr77" (OuterVolumeSpecName: "kube-api-access-cgr77") pod "ff7ec9fa-b726-42d3-979c-2ba885817bfa" (UID: "ff7ec9fa-b726-42d3-979c-2ba885817bfa"). InnerVolumeSpecName "kube-api-access-cgr77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:25:19.868168 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:19.868145    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgr77\" (UniqueName: \"kubernetes.io/projected/ff7ec9fa-b726-42d3-979c-2ba885817bfa-kube-api-access-cgr77\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:25:19.868168 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:19.868167    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7ec9fa-b726-42d3-979c-2ba885817bfa-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:25:20.089048 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:20.088965    2559 generic.go:356] "Generic (PLEG): container finished" podID="ff7ec9fa-b726-42d3-979c-2ba885817bfa" containerID="ad47fa01a1aa4141bdf0978f5d556fbdbd09033cebd0b392086c7fd810a44b87" exitCode=0
Sep 08 10:25:20.089048 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:20.089014    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"
Sep 08 10:25:20.089280 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:20.089050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2" event={"ID":"ff7ec9fa-b726-42d3-979c-2ba885817bfa","Type":"ContainerDied","Data":"ad47fa01a1aa4141bdf0978f5d556fbdbd09033cebd0b392086c7fd810a44b87"}
Sep 08 10:25:20.089280 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:20.089103    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2" event={"ID":"ff7ec9fa-b726-42d3-979c-2ba885817bfa","Type":"ContainerDied","Data":"9c5849d807117b3e7416cc3978e42d85125c85d9d9c4fb2129df4e775399f684"}
Sep 08 10:25:20.089280 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:20.089126    2559 scope.go:117] "RemoveContainer" containerID="ad47fa01a1aa4141bdf0978f5d556fbdbd09033cebd0b392086c7fd810a44b87"
Sep 08 10:25:20.098168 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:20.098153    2559 scope.go:117] "RemoveContainer" containerID="ad47fa01a1aa4141bdf0978f5d556fbdbd09033cebd0b392086c7fd810a44b87"
Sep 08 10:25:20.098457 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:25:20.098440    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad47fa01a1aa4141bdf0978f5d556fbdbd09033cebd0b392086c7fd810a44b87\": container with ID starting with ad47fa01a1aa4141bdf0978f5d556fbdbd09033cebd0b392086c7fd810a44b87 not found: ID does not exist" containerID="ad47fa01a1aa4141bdf0978f5d556fbdbd09033cebd0b392086c7fd810a44b87"
Sep 08 10:25:20.098513 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:20.098464    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad47fa01a1aa4141bdf0978f5d556fbdbd09033cebd0b392086c7fd810a44b87"} err="failed to get container status \"ad47fa01a1aa4141bdf0978f5d556fbdbd09033cebd0b392086c7fd810a44b87\": rpc error: code = NotFound desc = could not find container \"ad47fa01a1aa4141bdf0978f5d556fbdbd09033cebd0b392086c7fd810a44b87\": container with ID starting with ad47fa01a1aa4141bdf0978f5d556fbdbd09033cebd0b392086c7fd810a44b87 not found: ID does not exist"
Sep 08 10:25:20.110206 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:20.110184    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"]
Sep 08 10:25:20.116359 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:20.116337    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-22af43-78b7db4d79-jsjx2"]
Sep 08 10:25:20.637395 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:20.637368    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3942bc46-55c0-4b5a-b0c6-9c4880709b7e" path="/var/lib/kubelet/pods/3942bc46-55c0-4b5a-b0c6-9c4880709b7e/volumes"
Sep 08 10:25:20.637795 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:20.637779    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff7ec9fa-b726-42d3-979c-2ba885817bfa" path="/var/lib/kubelet/pods/ff7ec9fa-b726-42d3-979c-2ba885817bfa/volumes"
Sep 08 10:25:20.916753 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:20.916728    2559 scope.go:117] "RemoveContainer" containerID="3aca9214ace35cc95a29e9694d6efe03628d00e55f3d70876d330aa43e8c5297"
Sep 08 10:25:26.437654 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.437611    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd"]
Sep 08 10:25:26.438231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.438210    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff7ec9fa-b726-42d3-979c-2ba885817bfa" containerName="payload-processing"
Sep 08 10:25:26.438372 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.438233    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff7ec9fa-b726-42d3-979c-2ba885817bfa" containerName="payload-processing"
Sep 08 10:25:26.438372 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.438307    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3942bc46-55c0-4b5a-b0c6-9c4880709b7e" containerName="revoke-keys"
Sep 08 10:25:26.438372 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.438318    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3942bc46-55c0-4b5a-b0c6-9c4880709b7e" containerName="revoke-keys"
Sep 08 10:25:26.438526 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.438422    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff7ec9fa-b726-42d3-979c-2ba885817bfa" containerName="payload-processing"
Sep 08 10:25:26.438526 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.438435    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3942bc46-55c0-4b5a-b0c6-9c4880709b7e" containerName="revoke-keys"
Sep 08 10:25:26.443107 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.443085    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd"
Sep 08 10:25:26.445984 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.445966    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:25:26.446049 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.445967    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:25:26.450685 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.450658    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd"]
Sep 08 10:25:26.525056 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.525030    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639f357c-0e9b-4023-b0be-bf738aefd2da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd\" (UID: \"639f357c-0e9b-4023-b0be-bf738aefd2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd"
Sep 08 10:25:26.626171 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.626143    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639f357c-0e9b-4023-b0be-bf738aefd2da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd\" (UID: \"639f357c-0e9b-4023-b0be-bf738aefd2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd"
Sep 08 10:25:26.626782 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.626762    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639f357c-0e9b-4023-b0be-bf738aefd2da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd\" (UID: \"639f357c-0e9b-4023-b0be-bf738aefd2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd"
Sep 08 10:25:26.755918 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:26.755843    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd"
Sep 08 10:25:27.031165 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:27.031139    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd"]
Sep 08 10:25:27.033839 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:25:27.033812    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639f357c_0e9b_4023_b0be_bf738aefd2da.slice/crio-066b123982bfcf99ee7155d6d31a8ed73ce5886a6bf246300387e313d4892441 WatchSource:0}: Error finding container 066b123982bfcf99ee7155d6d31a8ed73ce5886a6bf246300387e313d4892441: Status 404 returned error can't find the container with id 066b123982bfcf99ee7155d6d31a8ed73ce5886a6bf246300387e313d4892441
Sep 08 10:25:27.134130 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:27.134104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd" event={"ID":"639f357c-0e9b-4023-b0be-bf738aefd2da","Type":"ContainerStarted","Data":"066b123982bfcf99ee7155d6d31a8ed73ce5886a6bf246300387e313d4892441"}
Sep 08 10:25:28.145018 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:28.144936    2559 generic.go:356] "Generic (PLEG): container finished" podID="639f357c-0e9b-4023-b0be-bf738aefd2da" containerID="bf7f2acf6448fe3c3793599713c43e94b556fb41e1e59426b44b4443c817ffa0" exitCode=0
Sep 08 10:25:28.145413 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:28.145022    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd" event={"ID":"639f357c-0e9b-4023-b0be-bf738aefd2da","Type":"ContainerDied","Data":"bf7f2acf6448fe3c3793599713c43e94b556fb41e1e59426b44b4443c817ffa0"}
Sep 08 10:25:29.286538 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:29.286515    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd"
Sep 08 10:25:29.351916 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:29.351890    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639f357c-0e9b-4023-b0be-bf738aefd2da-maas-api-service-ca\") pod \"639f357c-0e9b-4023-b0be-bf738aefd2da\" (UID: \"639f357c-0e9b-4023-b0be-bf738aefd2da\") "
Sep 08 10:25:29.352302 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:29.352282    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639f357c-0e9b-4023-b0be-bf738aefd2da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "639f357c-0e9b-4023-b0be-bf738aefd2da" (UID: "639f357c-0e9b-4023-b0be-bf738aefd2da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:25:29.453055 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:29.453027    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639f357c-0e9b-4023-b0be-bf738aefd2da-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:25:30.157147 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:30.157120    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd"
Sep 08 10:25:30.157147 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:30.157130    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd" event={"ID":"639f357c-0e9b-4023-b0be-bf738aefd2da","Type":"ContainerDied","Data":"066b123982bfcf99ee7155d6d31a8ed73ce5886a6bf246300387e313d4892441"}
Sep 08 10:25:30.157354 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:30.157175    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="066b123982bfcf99ee7155d6d31a8ed73ce5886a6bf246300387e313d4892441"
Sep 08 10:25:38.196750 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:38.196706    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"]
Sep 08 10:25:41.948099 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:41.948064    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"]
Sep 08 10:25:41.948523 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:41.948284    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp" podUID="b27ddf73-94dd-4424-a19b-2c6889b41359" containerName="payload-processing" containerID="cri-o://cacd494bd8dc10cd1051e125c53d1410a15208cda6a1827c5d26ef8dffa0abbf" gracePeriod=30
Sep 08 10:25:42.117462 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.117417    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd"]
Sep 08 10:25:42.120959 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.120936    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e30f6-qdzmd"]
Sep 08 10:25:42.214921 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.214898    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"
Sep 08 10:25:42.224345 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.224317    2559 generic.go:356] "Generic (PLEG): container finished" podID="b27ddf73-94dd-4424-a19b-2c6889b41359" containerID="cacd494bd8dc10cd1051e125c53d1410a15208cda6a1827c5d26ef8dffa0abbf" exitCode=0
Sep 08 10:25:42.224479 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.224349    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"
Sep 08 10:25:42.224479 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.224369    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp" event={"ID":"b27ddf73-94dd-4424-a19b-2c6889b41359","Type":"ContainerDied","Data":"cacd494bd8dc10cd1051e125c53d1410a15208cda6a1827c5d26ef8dffa0abbf"}
Sep 08 10:25:42.224479 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.224434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp" event={"ID":"b27ddf73-94dd-4424-a19b-2c6889b41359","Type":"ContainerDied","Data":"07fda58c252ea80a14ec9b23edf3aca177af876779bf0801d90b3d54e49e7c68"}
Sep 08 10:25:42.224479 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.224465    2559 scope.go:117] "RemoveContainer" containerID="cacd494bd8dc10cd1051e125c53d1410a15208cda6a1827c5d26ef8dffa0abbf"
Sep 08 10:25:42.234657 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.234636    2559 scope.go:117] "RemoveContainer" containerID="cacd494bd8dc10cd1051e125c53d1410a15208cda6a1827c5d26ef8dffa0abbf"
Sep 08 10:25:42.234993 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:25:42.234974    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cacd494bd8dc10cd1051e125c53d1410a15208cda6a1827c5d26ef8dffa0abbf\": container with ID starting with cacd494bd8dc10cd1051e125c53d1410a15208cda6a1827c5d26ef8dffa0abbf not found: ID does not exist" containerID="cacd494bd8dc10cd1051e125c53d1410a15208cda6a1827c5d26ef8dffa0abbf"
Sep 08 10:25:42.235060 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.235002    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cacd494bd8dc10cd1051e125c53d1410a15208cda6a1827c5d26ef8dffa0abbf"} err="failed to get container status \"cacd494bd8dc10cd1051e125c53d1410a15208cda6a1827c5d26ef8dffa0abbf\": rpc error: code = NotFound desc = could not find container \"cacd494bd8dc10cd1051e125c53d1410a15208cda6a1827c5d26ef8dffa0abbf\": container with ID starting with cacd494bd8dc10cd1051e125c53d1410a15208cda6a1827c5d26ef8dffa0abbf not found: ID does not exist"
Sep 08 10:25:42.362274 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.362234    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27ddf73-94dd-4424-a19b-2c6889b41359-plugins-config-volume\") pod \"b27ddf73-94dd-4424-a19b-2c6889b41359\" (UID: \"b27ddf73-94dd-4424-a19b-2c6889b41359\") "
Sep 08 10:25:42.362391 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.362377    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54g7z\" (UniqueName: \"kubernetes.io/projected/b27ddf73-94dd-4424-a19b-2c6889b41359-kube-api-access-54g7z\") pod \"b27ddf73-94dd-4424-a19b-2c6889b41359\" (UID: \"b27ddf73-94dd-4424-a19b-2c6889b41359\") "
Sep 08 10:25:42.362649 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.362629    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27ddf73-94dd-4424-a19b-2c6889b41359-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b27ddf73-94dd-4424-a19b-2c6889b41359" (UID: "b27ddf73-94dd-4424-a19b-2c6889b41359"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:25:42.362699 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.362688    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27ddf73-94dd-4424-a19b-2c6889b41359-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:25:42.364532 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.364508    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27ddf73-94dd-4424-a19b-2c6889b41359-kube-api-access-54g7z" (OuterVolumeSpecName: "kube-api-access-54g7z") pod "b27ddf73-94dd-4424-a19b-2c6889b41359" (UID: "b27ddf73-94dd-4424-a19b-2c6889b41359"). InnerVolumeSpecName "kube-api-access-54g7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:25:42.463446 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.463388    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54g7z\" (UniqueName: \"kubernetes.io/projected/b27ddf73-94dd-4424-a19b-2c6889b41359-kube-api-access-54g7z\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:25:42.553598 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.553576    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"]
Sep 08 10:25:42.557177 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.557156    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9e30f6-578494498c-lkxbp"]
Sep 08 10:25:42.637399 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.637375    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="639f357c-0e9b-4023-b0be-bf738aefd2da" path="/var/lib/kubelet/pods/639f357c-0e9b-4023-b0be-bf738aefd2da/volumes"
Sep 08 10:25:42.637771 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:42.637757    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b27ddf73-94dd-4424-a19b-2c6889b41359" path="/var/lib/kubelet/pods/b27ddf73-94dd-4424-a19b-2c6889b41359/volumes"
Sep 08 10:25:49.036116 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.036070    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w"]
Sep 08 10:25:49.036550 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.036534    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b27ddf73-94dd-4424-a19b-2c6889b41359" containerName="payload-processing"
Sep 08 10:25:49.036550 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.036550    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b27ddf73-94dd-4424-a19b-2c6889b41359" containerName="payload-processing"
Sep 08 10:25:49.036628 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.036560    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="639f357c-0e9b-4023-b0be-bf738aefd2da" containerName="revoke-keys"
Sep 08 10:25:49.036628 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.036566    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="639f357c-0e9b-4023-b0be-bf738aefd2da" containerName="revoke-keys"
Sep 08 10:25:49.036696 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.036641    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b27ddf73-94dd-4424-a19b-2c6889b41359" containerName="payload-processing"
Sep 08 10:25:49.036696 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.036651    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="639f357c-0e9b-4023-b0be-bf738aefd2da" containerName="revoke-keys"
Sep 08 10:25:49.041287 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.041269    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w"
Sep 08 10:25:49.044610 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.044580    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:25:49.045009 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.044986    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:25:49.050536 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.050515    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w"]
Sep 08 10:25:49.121192 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.121167    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952f152d-79bd-476e-8841-a53deddf49f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w\" (UID: \"952f152d-79bd-476e-8841-a53deddf49f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w"
Sep 08 10:25:49.221615 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.221585    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952f152d-79bd-476e-8841-a53deddf49f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w\" (UID: \"952f152d-79bd-476e-8841-a53deddf49f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w"
Sep 08 10:25:49.222179 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.222160    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952f152d-79bd-476e-8841-a53deddf49f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w\" (UID: \"952f152d-79bd-476e-8841-a53deddf49f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w"
Sep 08 10:25:49.357031 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.356961    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w"
Sep 08 10:25:49.631746 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:49.631715    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w"]
Sep 08 10:25:49.633819 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:25:49.633796    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952f152d_79bd_476e_8841_a53deddf49f9.slice/crio-271d0959982005d061b8e440e30c6a8b63bad54ee50836bd4481a0a6c4e3229e WatchSource:0}: Error finding container 271d0959982005d061b8e440e30c6a8b63bad54ee50836bd4481a0a6c4e3229e: Status 404 returned error can't find the container with id 271d0959982005d061b8e440e30c6a8b63bad54ee50836bd4481a0a6c4e3229e
Sep 08 10:25:50.272638 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:50.272608    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w" event={"ID":"952f152d-79bd-476e-8841-a53deddf49f9","Type":"ContainerStarted","Data":"f8311f1a729664faa081dccab4bba56c335fd2899227665bf7aac691e213aac3"}
Sep 08 10:25:50.273038 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:50.272660    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w" event={"ID":"952f152d-79bd-476e-8841-a53deddf49f9","Type":"ContainerStarted","Data":"271d0959982005d061b8e440e30c6a8b63bad54ee50836bd4481a0a6c4e3229e"}
Sep 08 10:25:50.287826 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:50.287783    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w" podStartSLOduration=1.287767365 podStartE2EDuration="1.287767365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:25:50.28747535 +0000 UTC m=+1470.128699755" watchObservedRunningTime="2026-09-08 10:25:50.287767365 +0000 UTC m=+1470.128991747"
Sep 08 10:25:51.279169 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:51.279133    2559 generic.go:356] "Generic (PLEG): container finished" podID="952f152d-79bd-476e-8841-a53deddf49f9" containerID="f8311f1a729664faa081dccab4bba56c335fd2899227665bf7aac691e213aac3" exitCode=0
Sep 08 10:25:51.279565 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:51.279218    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w" event={"ID":"952f152d-79bd-476e-8841-a53deddf49f9","Type":"ContainerDied","Data":"f8311f1a729664faa081dccab4bba56c335fd2899227665bf7aac691e213aac3"}
Sep 08 10:25:52.424986 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:52.424957    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w"
Sep 08 10:25:52.549721 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:52.549643    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952f152d-79bd-476e-8841-a53deddf49f9-maas-api-service-ca\") pod \"952f152d-79bd-476e-8841-a53deddf49f9\" (UID: \"952f152d-79bd-476e-8841-a53deddf49f9\") "
Sep 08 10:25:52.550003 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:52.549982    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952f152d-79bd-476e-8841-a53deddf49f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "952f152d-79bd-476e-8841-a53deddf49f9" (UID: "952f152d-79bd-476e-8841-a53deddf49f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:25:52.650839 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:52.650820    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952f152d-79bd-476e-8841-a53deddf49f9-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:25:53.216629 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.216582    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf" podUID="01c2011f-4ab1-471d-86b1-40bffa377bb1" containerName="main" containerID="cri-o://e04afa955a2cfe25f3b4aa70534f082743dab37d037df54f56ea231ccb452743" gracePeriod=45
Sep 08 10:25:53.291109 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.291086    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w"
Sep 08 10:25:53.291231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.291127    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w" event={"ID":"952f152d-79bd-476e-8841-a53deddf49f9","Type":"ContainerDied","Data":"271d0959982005d061b8e440e30c6a8b63bad54ee50836bd4481a0a6c4e3229e"}
Sep 08 10:25:53.291231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.291172    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="271d0959982005d061b8e440e30c6a8b63bad54ee50836bd4481a0a6c4e3229e"
Sep 08 10:25:53.467039 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.466983    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:25:53.553475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.553448    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"]
Sep 08 10:25:53.553947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.553923    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c2011f-4ab1-471d-86b1-40bffa377bb1" containerName="main"
Sep 08 10:25:53.553947 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.553945    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c2011f-4ab1-471d-86b1-40bffa377bb1" containerName="main"
Sep 08 10:25:53.554138 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.553967    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="952f152d-79bd-476e-8841-a53deddf49f9" containerName="revoke-keys"
Sep 08 10:25:53.554138 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.553973    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="952f152d-79bd-476e-8841-a53deddf49f9" containerName="revoke-keys"
Sep 08 10:25:53.554138 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.554048    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="952f152d-79bd-476e-8841-a53deddf49f9" containerName="revoke-keys"
Sep 08 10:25:53.554138 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.554062    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="01c2011f-4ab1-471d-86b1-40bffa377bb1" containerName="main"
Sep 08 10:25:53.556800 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.556782    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.558706 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.558687    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-tmp-dir\") pod \"01c2011f-4ab1-471d-86b1-40bffa377bb1\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") "
Sep 08 10:25:53.558818 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.558721    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbgd4\" (UniqueName: \"kubernetes.io/projected/01c2011f-4ab1-471d-86b1-40bffa377bb1-kube-api-access-cbgd4\") pod \"01c2011f-4ab1-471d-86b1-40bffa377bb1\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") "
Sep 08 10:25:53.558818 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.558785    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-model-cache\") pod \"01c2011f-4ab1-471d-86b1-40bffa377bb1\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") "
Sep 08 10:25:53.558937 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.558829    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c2011f-4ab1-471d-86b1-40bffa377bb1-tls-certs\") pod \"01c2011f-4ab1-471d-86b1-40bffa377bb1\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") "
Sep 08 10:25:53.558937 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.558862    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-dshm\") pod \"01c2011f-4ab1-471d-86b1-40bffa377bb1\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") "
Sep 08 10:25:53.558937 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.558931    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-home\") pod \"01c2011f-4ab1-471d-86b1-40bffa377bb1\" (UID: \"01c2011f-4ab1-471d-86b1-40bffa377bb1\") "
Sep 08 10:25:53.559092 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.558973    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01c2011f-4ab1-471d-86b1-40bffa377bb1" (UID: "01c2011f-4ab1-471d-86b1-40bffa377bb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:25:53.559092 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.559006    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-model-cache" (OuterVolumeSpecName: "model-cache") pod "01c2011f-4ab1-471d-86b1-40bffa377bb1" (UID: "01c2011f-4ab1-471d-86b1-40bffa377bb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:25:53.559281 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.559235    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-model-cache\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:25:53.559444 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.559424    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-tmp-dir\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:25:53.559444 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.559273    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-home" (OuterVolumeSpecName: "home") pod "01c2011f-4ab1-471d-86b1-40bffa377bb1" (UID: "01c2011f-4ab1-471d-86b1-40bffa377bb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:25:53.561107 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.561079    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c2011f-4ab1-471d-86b1-40bffa377bb1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01c2011f-4ab1-471d-86b1-40bffa377bb1" (UID: "01c2011f-4ab1-471d-86b1-40bffa377bb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:25:53.561363 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.561341    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-dshm" (OuterVolumeSpecName: "dshm") pod "01c2011f-4ab1-471d-86b1-40bffa377bb1" (UID: "01c2011f-4ab1-471d-86b1-40bffa377bb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:25:53.561363 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.561351    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c2011f-4ab1-471d-86b1-40bffa377bb1-kube-api-access-cbgd4" (OuterVolumeSpecName: "kube-api-access-cbgd4") pod "01c2011f-4ab1-471d-86b1-40bffa377bb1" (UID: "01c2011f-4ab1-471d-86b1-40bffa377bb1"). InnerVolumeSpecName "kube-api-access-cbgd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:25:53.562953 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.562933    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-71a7a1\"/\"rate-test-model-71a7a1-kserve-self-signed-certs\""
Sep 08 10:25:53.571716 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.571694    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"]
Sep 08 10:25:53.660055 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.660023    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-model-cache\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.660181 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.660078    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-dshm\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.660181 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.660159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef733af-b8be-4d3b-b671-49534e93b477-tls-certs\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.660306 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.660219    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-tmp-dir\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.660306 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.660301    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-home\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.660392 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.660331    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzcpf\" (UniqueName: \"kubernetes.io/projected/3ef733af-b8be-4d3b-b671-49534e93b477-kube-api-access-pzcpf\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.660392 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.660373    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-dshm\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:25:53.660392 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.660384    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c2011f-4ab1-471d-86b1-40bffa377bb1-home\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:25:53.660493 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.660393    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbgd4\" (UniqueName: \"kubernetes.io/projected/01c2011f-4ab1-471d-86b1-40bffa377bb1-kube-api-access-cbgd4\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:25:53.660493 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.660402    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c2011f-4ab1-471d-86b1-40bffa377bb1-tls-certs\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:25:53.761453 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.761384    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-dshm\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.761453 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.761424    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef733af-b8be-4d3b-b671-49534e93b477-tls-certs\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.761453 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.761453    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-tmp-dir\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.761665 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.761489    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-home\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.762341 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.762047    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzcpf\" (UniqueName: \"kubernetes.io/projected/3ef733af-b8be-4d3b-b671-49534e93b477-kube-api-access-pzcpf\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.762341 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.762220    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-model-cache\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.762341 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.762286    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-tmp-dir\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.762581 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.762354    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-home\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.762710 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.762646    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-model-cache\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.770639 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.770615    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-dshm\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.772056 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.772030    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef733af-b8be-4d3b-b671-49534e93b477-tls-certs\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.773183 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.773160    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzcpf\" (UniqueName: \"kubernetes.io/projected/3ef733af-b8be-4d3b-b671-49534e93b477-kube-api-access-pzcpf\") pod \"rate-test-model-71a7a1-kserve-647ccfd99-h589h\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") " pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:53.876914 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:53.876883    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:25:54.151023 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:54.150996    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"]
Sep 08 10:25:54.153349 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:25:54.153321    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ef733af_b8be_4d3b_b671_49534e93b477.slice/crio-d8115cd16de9d952b0b5ce10181b71748c613dbb8b96dab8baea2a341d3cce82 WatchSource:0}: Error finding container d8115cd16de9d952b0b5ce10181b71748c613dbb8b96dab8baea2a341d3cce82: Status 404 returned error can't find the container with id d8115cd16de9d952b0b5ce10181b71748c613dbb8b96dab8baea2a341d3cce82
Sep 08 10:25:54.297934 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:54.297890    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h" event={"ID":"3ef733af-b8be-4d3b-b671-49534e93b477","Type":"ContainerStarted","Data":"2bbb6ed29c69a56040e395de99198dcebcf05584990e03a24bbb2d2f74a7643f"}
Sep 08 10:25:54.297934 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:54.297938    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h" event={"ID":"3ef733af-b8be-4d3b-b671-49534e93b477","Type":"ContainerStarted","Data":"d8115cd16de9d952b0b5ce10181b71748c613dbb8b96dab8baea2a341d3cce82"}
Sep 08 10:25:54.299366 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:54.299343    2559 generic.go:356] "Generic (PLEG): container finished" podID="01c2011f-4ab1-471d-86b1-40bffa377bb1" containerID="e04afa955a2cfe25f3b4aa70534f082743dab37d037df54f56ea231ccb452743" exitCode=0
Sep 08 10:25:54.299477 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:54.299393    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"
Sep 08 10:25:54.299477 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:54.299420    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf" event={"ID":"01c2011f-4ab1-471d-86b1-40bffa377bb1","Type":"ContainerDied","Data":"e04afa955a2cfe25f3b4aa70534f082743dab37d037df54f56ea231ccb452743"}
Sep 08 10:25:54.299477 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:54.299455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf" event={"ID":"01c2011f-4ab1-471d-86b1-40bffa377bb1","Type":"ContainerDied","Data":"c17ba05da3a94ebba4fcabcf2760d4aa3766e089d405932db6a0260f83129bed"}
Sep 08 10:25:54.299477 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:54.299472    2559 scope.go:117] "RemoveContainer" containerID="e04afa955a2cfe25f3b4aa70534f082743dab37d037df54f56ea231ccb452743"
Sep 08 10:25:54.308682 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:54.308666    2559 scope.go:117] "RemoveContainer" containerID="e04afa955a2cfe25f3b4aa70534f082743dab37d037df54f56ea231ccb452743"
Sep 08 10:25:54.308939 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:25:54.308922    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04afa955a2cfe25f3b4aa70534f082743dab37d037df54f56ea231ccb452743\": container with ID starting with e04afa955a2cfe25f3b4aa70534f082743dab37d037df54f56ea231ccb452743 not found: ID does not exist" containerID="e04afa955a2cfe25f3b4aa70534f082743dab37d037df54f56ea231ccb452743"
Sep 08 10:25:54.308979 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:54.308948    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04afa955a2cfe25f3b4aa70534f082743dab37d037df54f56ea231ccb452743"} err="failed to get container status \"e04afa955a2cfe25f3b4aa70534f082743dab37d037df54f56ea231ccb452743\": rpc error: code = NotFound desc = could not find container \"e04afa955a2cfe25f3b4aa70534f082743dab37d037df54f56ea231ccb452743\": container with ID starting with e04afa955a2cfe25f3b4aa70534f082743dab37d037df54f56ea231ccb452743 not found: ID does not exist"
Sep 08 10:25:54.317442 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:54.317402    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h" podStartSLOduration=1.317389762 podStartE2EDuration="1.317389762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:25:54.315759233 +0000 UTC m=+1474.156983619" watchObservedRunningTime="2026-09-08 10:25:54.317389762 +0000 UTC m=+1474.158614144"
Sep 08 10:25:54.331035 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:54.331014    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"]
Sep 08 10:25:54.336192 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:54.336172    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/sub-test-model-71a7a1-kserve-7bc55546dc-q8rvf"]
Sep 08 10:25:54.637331 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:54.637299    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01c2011f-4ab1-471d-86b1-40bffa377bb1" path="/var/lib/kubelet/pods/01c2011f-4ab1-471d-86b1-40bffa377bb1/volumes"
Sep 08 10:25:59.564100 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:59.564068    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"]
Sep 08 10:25:59.564519 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:59.564332    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj" podUID="b1b75c89-93d4-42fe-b1ba-c035e219439a" containerName="payload-processing" containerID="cri-o://32f7e2fc987afe337f99d4a2e1b3566cbcdd81946c34716e93e627b148ee7b9e" gracePeriod=30
Sep 08 10:25:59.570864 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:59.570840    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"]
Sep 08 10:25:59.571337 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:59.571309    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g" podUID="ed7bf9bb-5ff5-4a80-bf42-3f588e913045" containerName="payload-pre-processing" containerID="cri-o://2884024e0751ea345b660347dc6ff47552e6b62f49b04337cb75bbc41949922c" gracePeriod=30
Sep 08 10:25:59.724382 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:59.724338    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w"]
Sep 08 10:25:59.727028 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:59.726999    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-af58b2-8hx6w"]
Sep 08 10:25:59.848350 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:59.848321    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"
Sep 08 10:25:59.850721 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:25:59.850704    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"
Sep 08 10:26:00.017210 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.017186    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd58x\" (UniqueName: \"kubernetes.io/projected/ed7bf9bb-5ff5-4a80-bf42-3f588e913045-kube-api-access-wd58x\") pod \"ed7bf9bb-5ff5-4a80-bf42-3f588e913045\" (UID: \"ed7bf9bb-5ff5-4a80-bf42-3f588e913045\") "
Sep 08 10:26:00.017380 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.017227    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7bf9bb-5ff5-4a80-bf42-3f588e913045-plugins-config-volume\") pod \"ed7bf9bb-5ff5-4a80-bf42-3f588e913045\" (UID: \"ed7bf9bb-5ff5-4a80-bf42-3f588e913045\") "
Sep 08 10:26:00.017380 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.017304    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b75c89-93d4-42fe-b1ba-c035e219439a-plugins-config-volume\") pod \"b1b75c89-93d4-42fe-b1ba-c035e219439a\" (UID: \"b1b75c89-93d4-42fe-b1ba-c035e219439a\") "
Sep 08 10:26:00.017380 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.017329    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c84l8\" (UniqueName: \"kubernetes.io/projected/b1b75c89-93d4-42fe-b1ba-c035e219439a-kube-api-access-c84l8\") pod \"b1b75c89-93d4-42fe-b1ba-c035e219439a\" (UID: \"b1b75c89-93d4-42fe-b1ba-c035e219439a\") "
Sep 08 10:26:00.017660 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.017637    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7bf9bb-5ff5-4a80-bf42-3f588e913045-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed7bf9bb-5ff5-4a80-bf42-3f588e913045" (UID: "ed7bf9bb-5ff5-4a80-bf42-3f588e913045"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:26:00.017720 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.017672    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b75c89-93d4-42fe-b1ba-c035e219439a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1b75c89-93d4-42fe-b1ba-c035e219439a" (UID: "b1b75c89-93d4-42fe-b1ba-c035e219439a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:26:00.019370 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.019346    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7bf9bb-5ff5-4a80-bf42-3f588e913045-kube-api-access-wd58x" (OuterVolumeSpecName: "kube-api-access-wd58x") pod "ed7bf9bb-5ff5-4a80-bf42-3f588e913045" (UID: "ed7bf9bb-5ff5-4a80-bf42-3f588e913045"). InnerVolumeSpecName "kube-api-access-wd58x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:26:00.019498 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.019475    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b75c89-93d4-42fe-b1ba-c035e219439a-kube-api-access-c84l8" (OuterVolumeSpecName: "kube-api-access-c84l8") pod "b1b75c89-93d4-42fe-b1ba-c035e219439a" (UID: "b1b75c89-93d4-42fe-b1ba-c035e219439a"). InnerVolumeSpecName "kube-api-access-c84l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:26:00.118810 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.118741    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd58x\" (UniqueName: \"kubernetes.io/projected/ed7bf9bb-5ff5-4a80-bf42-3f588e913045-kube-api-access-wd58x\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:26:00.118810 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.118765    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7bf9bb-5ff5-4a80-bf42-3f588e913045-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:26:00.118810 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.118774    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b75c89-93d4-42fe-b1ba-c035e219439a-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:26:00.118810 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.118785    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c84l8\" (UniqueName: \"kubernetes.io/projected/b1b75c89-93d4-42fe-b1ba-c035e219439a-kube-api-access-c84l8\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:26:00.332737 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.332704    2559 generic.go:356] "Generic (PLEG): container finished" podID="ed7bf9bb-5ff5-4a80-bf42-3f588e913045" containerID="2884024e0751ea345b660347dc6ff47552e6b62f49b04337cb75bbc41949922c" exitCode=0
Sep 08 10:26:00.332907 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.332746    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"
Sep 08 10:26:00.332907 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.332750    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g" event={"ID":"ed7bf9bb-5ff5-4a80-bf42-3f588e913045","Type":"ContainerDied","Data":"2884024e0751ea345b660347dc6ff47552e6b62f49b04337cb75bbc41949922c"}
Sep 08 10:26:00.332907 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.332827    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g" event={"ID":"ed7bf9bb-5ff5-4a80-bf42-3f588e913045","Type":"ContainerDied","Data":"c2477a96154d63da534e168ccf6a70ece78ed75c2d306ea4202fb7a7abc7e9cf"}
Sep 08 10:26:00.332907 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.332851    2559 scope.go:117] "RemoveContainer" containerID="2884024e0751ea345b660347dc6ff47552e6b62f49b04337cb75bbc41949922c"
Sep 08 10:26:00.334145 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.334125    2559 generic.go:356] "Generic (PLEG): container finished" podID="b1b75c89-93d4-42fe-b1ba-c035e219439a" containerID="32f7e2fc987afe337f99d4a2e1b3566cbcdd81946c34716e93e627b148ee7b9e" exitCode=0
Sep 08 10:26:00.334284 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.334169    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"
Sep 08 10:26:00.334284 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.334195    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj" event={"ID":"b1b75c89-93d4-42fe-b1ba-c035e219439a","Type":"ContainerDied","Data":"32f7e2fc987afe337f99d4a2e1b3566cbcdd81946c34716e93e627b148ee7b9e"}
Sep 08 10:26:00.334284 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.334226    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj" event={"ID":"b1b75c89-93d4-42fe-b1ba-c035e219439a","Type":"ContainerDied","Data":"ffaeab56982a630b6e0059ba47b3bea6386420abf251f777ca606db90d2ccd35"}
Sep 08 10:26:00.342670 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.342654    2559 scope.go:117] "RemoveContainer" containerID="2884024e0751ea345b660347dc6ff47552e6b62f49b04337cb75bbc41949922c"
Sep 08 10:26:00.342902 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:26:00.342881    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2884024e0751ea345b660347dc6ff47552e6b62f49b04337cb75bbc41949922c\": container with ID starting with 2884024e0751ea345b660347dc6ff47552e6b62f49b04337cb75bbc41949922c not found: ID does not exist" containerID="2884024e0751ea345b660347dc6ff47552e6b62f49b04337cb75bbc41949922c"
Sep 08 10:26:00.342993 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.342908    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2884024e0751ea345b660347dc6ff47552e6b62f49b04337cb75bbc41949922c"} err="failed to get container status \"2884024e0751ea345b660347dc6ff47552e6b62f49b04337cb75bbc41949922c\": rpc error: code = NotFound desc = could not find container \"2884024e0751ea345b660347dc6ff47552e6b62f49b04337cb75bbc41949922c\": container with ID starting with 2884024e0751ea345b660347dc6ff47552e6b62f49b04337cb75bbc41949922c not found: ID does not exist"
Sep 08 10:26:00.342993 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.342924    2559 scope.go:117] "RemoveContainer" containerID="32f7e2fc987afe337f99d4a2e1b3566cbcdd81946c34716e93e627b148ee7b9e"
Sep 08 10:26:00.353293 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.353024    2559 scope.go:117] "RemoveContainer" containerID="32f7e2fc987afe337f99d4a2e1b3566cbcdd81946c34716e93e627b148ee7b9e"
Sep 08 10:26:00.353604 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:26:00.353571    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f7e2fc987afe337f99d4a2e1b3566cbcdd81946c34716e93e627b148ee7b9e\": container with ID starting with 32f7e2fc987afe337f99d4a2e1b3566cbcdd81946c34716e93e627b148ee7b9e not found: ID does not exist" containerID="32f7e2fc987afe337f99d4a2e1b3566cbcdd81946c34716e93e627b148ee7b9e"
Sep 08 10:26:00.353703 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.353612    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f7e2fc987afe337f99d4a2e1b3566cbcdd81946c34716e93e627b148ee7b9e"} err="failed to get container status \"32f7e2fc987afe337f99d4a2e1b3566cbcdd81946c34716e93e627b148ee7b9e\": rpc error: code = NotFound desc = could not find container \"32f7e2fc987afe337f99d4a2e1b3566cbcdd81946c34716e93e627b148ee7b9e\": container with ID starting with 32f7e2fc987afe337f99d4a2e1b3566cbcdd81946c34716e93e627b148ee7b9e not found: ID does not exist"
Sep 08 10:26:00.355479 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.355461    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"]
Sep 08 10:26:00.360581 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.360561    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-af58b2-85dd7c9fd-mnp9g"]
Sep 08 10:26:00.372437 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.372382    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"]
Sep 08 10:26:00.377311 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.377289    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-af58b2-5888bb6ff7-hjmnj"]
Sep 08 10:26:00.636957 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.636888    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="952f152d-79bd-476e-8841-a53deddf49f9" path="/var/lib/kubelet/pods/952f152d-79bd-476e-8841-a53deddf49f9/volumes"
Sep 08 10:26:00.637337 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.637274    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1b75c89-93d4-42fe-b1ba-c035e219439a" path="/var/lib/kubelet/pods/b1b75c89-93d4-42fe-b1ba-c035e219439a/volumes"
Sep 08 10:26:00.637596 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:00.637583    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed7bf9bb-5ff5-4a80-bf42-3f588e913045" path="/var/lib/kubelet/pods/ed7bf9bb-5ff5-4a80-bf42-3f588e913045/volumes"
Sep 08 10:26:03.877332 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:03.877301    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:26:03.877688 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:03.877348    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:26:03.890094 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:03.890070    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:26:04.370554 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:04.370523    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:26:06.575063 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.575029    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h"]
Sep 08 10:26:06.575483 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.575469    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1b75c89-93d4-42fe-b1ba-c035e219439a" containerName="payload-processing"
Sep 08 10:26:06.575483 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.575484    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1b75c89-93d4-42fe-b1ba-c035e219439a" containerName="payload-processing"
Sep 08 10:26:06.575558 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.575495    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed7bf9bb-5ff5-4a80-bf42-3f588e913045" containerName="payload-pre-processing"
Sep 08 10:26:06.575558 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.575514    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed7bf9bb-5ff5-4a80-bf42-3f588e913045" containerName="payload-pre-processing"
Sep 08 10:26:06.575626 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.575575    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed7bf9bb-5ff5-4a80-bf42-3f588e913045" containerName="payload-pre-processing"
Sep 08 10:26:06.575626 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.575583    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1b75c89-93d4-42fe-b1ba-c035e219439a" containerName="payload-processing"
Sep 08 10:26:06.580066 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.580046    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h"
Sep 08 10:26:06.582910 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.582889    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:26:06.582910 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.582901    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:26:06.588470 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.588448    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h"]
Sep 08 10:26:06.769607 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.769577    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbba3be3-6d3d-467e-807d-186d7b9e5ea1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h\" (UID: \"fbba3be3-6d3d-467e-807d-186d7b9e5ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h"
Sep 08 10:26:06.870204 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.870130    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbba3be3-6d3d-467e-807d-186d7b9e5ea1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h\" (UID: \"fbba3be3-6d3d-467e-807d-186d7b9e5ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h"
Sep 08 10:26:06.870745 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.870725    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbba3be3-6d3d-467e-807d-186d7b9e5ea1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h\" (UID: \"fbba3be3-6d3d-467e-807d-186d7b9e5ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h"
Sep 08 10:26:06.892017 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:06.891995    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h"
Sep 08 10:26:07.165857 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:07.165829    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h"]
Sep 08 10:26:07.167769 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:26:07.167743    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbba3be3_6d3d_467e_807d_186d7b9e5ea1.slice/crio-d9f27da85f293393e3f3b3025a2cce867f9b6ba4af9385bb2e6208ec342aeab6 WatchSource:0}: Error finding container d9f27da85f293393e3f3b3025a2cce867f9b6ba4af9385bb2e6208ec342aeab6: Status 404 returned error can't find the container with id d9f27da85f293393e3f3b3025a2cce867f9b6ba4af9385bb2e6208ec342aeab6
Sep 08 10:26:07.376614 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:07.376583    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h" event={"ID":"fbba3be3-6d3d-467e-807d-186d7b9e5ea1","Type":"ContainerStarted","Data":"d9f27da85f293393e3f3b3025a2cce867f9b6ba4af9385bb2e6208ec342aeab6"}
Sep 08 10:26:08.384580 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:08.384548    2559 generic.go:356] "Generic (PLEG): container finished" podID="fbba3be3-6d3d-467e-807d-186d7b9e5ea1" containerID="b4ae0f989ade80f7c01a9a32125f6a1ff341b87200d62af31750c238e1bcdec9" exitCode=0
Sep 08 10:26:08.384975 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:08.384607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h" event={"ID":"fbba3be3-6d3d-467e-807d-186d7b9e5ea1","Type":"ContainerDied","Data":"b4ae0f989ade80f7c01a9a32125f6a1ff341b87200d62af31750c238e1bcdec9"}
Sep 08 10:26:09.524262 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:09.524217    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h"
Sep 08 10:26:09.592601 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:09.592571    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbba3be3-6d3d-467e-807d-186d7b9e5ea1-maas-api-service-ca\") pod \"fbba3be3-6d3d-467e-807d-186d7b9e5ea1\" (UID: \"fbba3be3-6d3d-467e-807d-186d7b9e5ea1\") "
Sep 08 10:26:09.592918 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:09.592897    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbba3be3-6d3d-467e-807d-186d7b9e5ea1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fbba3be3-6d3d-467e-807d-186d7b9e5ea1" (UID: "fbba3be3-6d3d-467e-807d-186d7b9e5ea1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:26:09.693462 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:09.693435    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbba3be3-6d3d-467e-807d-186d7b9e5ea1-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:26:10.397727 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:10.397703    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h"
Sep 08 10:26:10.397895 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:10.397741    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h" event={"ID":"fbba3be3-6d3d-467e-807d-186d7b9e5ea1","Type":"ContainerDied","Data":"d9f27da85f293393e3f3b3025a2cce867f9b6ba4af9385bb2e6208ec342aeab6"}
Sep 08 10:26:10.397895 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:10.397819    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9f27da85f293393e3f3b3025a2cce867f9b6ba4af9385bb2e6208ec342aeab6"
Sep 08 10:26:16.445905 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.445874    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"]
Sep 08 10:26:16.446511 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.446494    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbba3be3-6d3d-467e-807d-186d7b9e5ea1" containerName="revoke-keys"
Sep 08 10:26:16.446571 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.446514    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbba3be3-6d3d-467e-807d-186d7b9e5ea1" containerName="revoke-keys"
Sep 08 10:26:16.446607 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.446583    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbba3be3-6d3d-467e-807d-186d7b9e5ea1" containerName="revoke-keys"
Sep 08 10:26:16.449960 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.449936    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.458763 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.458732    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d6e53c\"/\"kube-root-ca.crt\""
Sep 08 10:26:16.459690 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.459662    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d6e53c\"/\"openshift-service-ca.crt\""
Sep 08 10:26:16.459904 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.459811    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d6e53c\"/\"default-dockercfg-spwd4\""
Sep 08 10:26:16.460106 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.460090    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d6e53c\"/\"rate-test-model-d6e53c-kserve-self-signed-certs\""
Sep 08 10:26:16.463069 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.463047    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"]
Sep 08 10:26:16.548578 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.548545    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-model-cache\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.548750 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.548598    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-dshm\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.548750 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.548655    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-tmp-dir\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.548750 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.548683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-home\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.548750 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.548722    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-tls-certs\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.548890 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.548761    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnxm5\" (UniqueName: \"kubernetes.io/projected/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-kube-api-access-gnxm5\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.650075 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.650045    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-model-cache\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.650222 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.650087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-dshm\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.650222 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.650111    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-tmp-dir\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.650222 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.650126    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-home\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.650222 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.650141    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-tls-certs\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.650222 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.650156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnxm5\" (UniqueName: \"kubernetes.io/projected/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-kube-api-access-gnxm5\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.650558 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.650528    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-tmp-dir\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.650558 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.650544    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-model-cache\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.650709 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.650621    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-home\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.652513 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.652488    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-dshm\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.652719 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.652704    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-tls-certs\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.658492 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.658471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnxm5\" (UniqueName: \"kubernetes.io/projected/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-kube-api-access-gnxm5\") pod \"rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") " pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:16.771512 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:16.771449    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:17.067528 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.067499    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"]
Sep 08 10:26:17.069982 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:26:17.069952    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf02bee9e_ed6f_4d12_9460_9aa36b42fe62.slice/crio-97ca086a9c4d0fe453cd7046848bf8cd0b492667c059408def9b0e2478999a0c WatchSource:0}: Error finding container 97ca086a9c4d0fe453cd7046848bf8cd0b492667c059408def9b0e2478999a0c: Status 404 returned error can't find the container with id 97ca086a9c4d0fe453cd7046848bf8cd0b492667c059408def9b0e2478999a0c
Sep 08 10:26:17.440126 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.440090    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp" event={"ID":"f02bee9e-ed6f-4d12-9460-9aa36b42fe62","Type":"ContainerStarted","Data":"51413d5a1e7d7f517e7b2f75e9ca3ed0c6b349fcf58dc58da8d6c9164a9097e2"}
Sep 08 10:26:17.440126 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.440139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp" event={"ID":"f02bee9e-ed6f-4d12-9460-9aa36b42fe62","Type":"ContainerStarted","Data":"97ca086a9c4d0fe453cd7046848bf8cd0b492667c059408def9b0e2478999a0c"}
Sep 08 10:26:17.463779 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.463722    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp" podStartSLOduration=1.463702867 podStartE2EDuration="1.463702867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:26:17.463651967 +0000 UTC m=+1497.304876351" watchObservedRunningTime="2026-09-08 10:26:17.463702867 +0000 UTC m=+1497.304927252"
Sep 08 10:26:17.536833 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.536802    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"]
Sep 08 10:26:17.537136 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.537002    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj" podUID="9d8e0fa7-f53d-400e-abcf-2b9f95fc0892" containerName="payload-processing" containerID="cri-o://0487aa3597290fab55bf5a415fe2a7b13549202656afb09cc72f09b46b796792" gracePeriod=30
Sep 08 10:26:17.723784 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.723705    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h"]
Sep 08 10:26:17.726723 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.726693    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ed05-dp65h"]
Sep 08 10:26:17.788587 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.788565    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"
Sep 08 10:26:17.863777 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.863751    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hz4p\" (UniqueName: \"kubernetes.io/projected/9d8e0fa7-f53d-400e-abcf-2b9f95fc0892-kube-api-access-9hz4p\") pod \"9d8e0fa7-f53d-400e-abcf-2b9f95fc0892\" (UID: \"9d8e0fa7-f53d-400e-abcf-2b9f95fc0892\") "
Sep 08 10:26:17.863912 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.863851    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8e0fa7-f53d-400e-abcf-2b9f95fc0892-plugins-config-volume\") pod \"9d8e0fa7-f53d-400e-abcf-2b9f95fc0892\" (UID: \"9d8e0fa7-f53d-400e-abcf-2b9f95fc0892\") "
Sep 08 10:26:17.864231 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.864209    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8e0fa7-f53d-400e-abcf-2b9f95fc0892-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d8e0fa7-f53d-400e-abcf-2b9f95fc0892" (UID: "9d8e0fa7-f53d-400e-abcf-2b9f95fc0892"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:26:17.865896 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.865868    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8e0fa7-f53d-400e-abcf-2b9f95fc0892-kube-api-access-9hz4p" (OuterVolumeSpecName: "kube-api-access-9hz4p") pod "9d8e0fa7-f53d-400e-abcf-2b9f95fc0892" (UID: "9d8e0fa7-f53d-400e-abcf-2b9f95fc0892"). InnerVolumeSpecName "kube-api-access-9hz4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:26:17.965650 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.965580    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hz4p\" (UniqueName: \"kubernetes.io/projected/9d8e0fa7-f53d-400e-abcf-2b9f95fc0892-kube-api-access-9hz4p\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:26:17.965650 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:17.965605    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8e0fa7-f53d-400e-abcf-2b9f95fc0892-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:26:18.447583 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:18.447549    2559 generic.go:356] "Generic (PLEG): container finished" podID="9d8e0fa7-f53d-400e-abcf-2b9f95fc0892" containerID="0487aa3597290fab55bf5a415fe2a7b13549202656afb09cc72f09b46b796792" exitCode=0
Sep 08 10:26:18.447726 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:18.447598    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"
Sep 08 10:26:18.447726 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:18.447634    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj" event={"ID":"9d8e0fa7-f53d-400e-abcf-2b9f95fc0892","Type":"ContainerDied","Data":"0487aa3597290fab55bf5a415fe2a7b13549202656afb09cc72f09b46b796792"}
Sep 08 10:26:18.447726 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:18.447685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj" event={"ID":"9d8e0fa7-f53d-400e-abcf-2b9f95fc0892","Type":"ContainerDied","Data":"d353ebf06175dbd7c01ee236beef83c7dd713515539a4053780e2dbff39335fa"}
Sep 08 10:26:18.447726 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:18.447708    2559 scope.go:117] "RemoveContainer" containerID="0487aa3597290fab55bf5a415fe2a7b13549202656afb09cc72f09b46b796792"
Sep 08 10:26:18.457322 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:18.457311    2559 scope.go:117] "RemoveContainer" containerID="0487aa3597290fab55bf5a415fe2a7b13549202656afb09cc72f09b46b796792"
Sep 08 10:26:18.457585 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:26:18.457567    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0487aa3597290fab55bf5a415fe2a7b13549202656afb09cc72f09b46b796792\": container with ID starting with 0487aa3597290fab55bf5a415fe2a7b13549202656afb09cc72f09b46b796792 not found: ID does not exist" containerID="0487aa3597290fab55bf5a415fe2a7b13549202656afb09cc72f09b46b796792"
Sep 08 10:26:18.457629 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:18.457593    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0487aa3597290fab55bf5a415fe2a7b13549202656afb09cc72f09b46b796792"} err="failed to get container status \"0487aa3597290fab55bf5a415fe2a7b13549202656afb09cc72f09b46b796792\": rpc error: code = NotFound desc = could not find container \"0487aa3597290fab55bf5a415fe2a7b13549202656afb09cc72f09b46b796792\": container with ID starting with 0487aa3597290fab55bf5a415fe2a7b13549202656afb09cc72f09b46b796792 not found: ID does not exist"
Sep 08 10:26:18.474078 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:18.474056    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"]
Sep 08 10:26:18.477894 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:18.477873    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37ed05-6964657655-wdfwj"]
Sep 08 10:26:18.636901 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:18.636875    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d8e0fa7-f53d-400e-abcf-2b9f95fc0892" path="/var/lib/kubelet/pods/9d8e0fa7-f53d-400e-abcf-2b9f95fc0892/volumes"
Sep 08 10:26:18.637216 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:18.637204    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbba3be3-6d3d-467e-807d-186d7b9e5ea1" path="/var/lib/kubelet/pods/fbba3be3-6d3d-467e-807d-186d7b9e5ea1/volumes"
Sep 08 10:26:20.725506 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:20.725479    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:26:20.727473 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:20.727453    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:26:26.771609 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:26.771576    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:26.772007 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:26.771635    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:26.784230 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:26.784206    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:26:27.508491 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:26:27.508461    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:27:02.741581 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:02.741542    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"]
Sep 08 10:27:03.735836 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:03.735802    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"]
Sep 08 10:27:17.760884 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:17.760826    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h" podUID="3ef733af-b8be-4d3b-b671-49534e93b477" containerName="main" containerID="cri-o://2bbb6ed29c69a56040e395de99198dcebcf05584990e03a24bbb2d2f74a7643f" gracePeriod=45
Sep 08 10:27:18.013601 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.013547    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:27:18.105025 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.104996    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-home\") pod \"3ef733af-b8be-4d3b-b671-49534e93b477\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") "
Sep 08 10:27:18.105173 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.105052    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef733af-b8be-4d3b-b671-49534e93b477-tls-certs\") pod \"3ef733af-b8be-4d3b-b671-49534e93b477\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") "
Sep 08 10:27:18.105173 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.105116    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzcpf\" (UniqueName: \"kubernetes.io/projected/3ef733af-b8be-4d3b-b671-49534e93b477-kube-api-access-pzcpf\") pod \"3ef733af-b8be-4d3b-b671-49534e93b477\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") "
Sep 08 10:27:18.105173 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.105144    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-tmp-dir\") pod \"3ef733af-b8be-4d3b-b671-49534e93b477\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") "
Sep 08 10:27:18.105173 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.105163    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-model-cache\") pod \"3ef733af-b8be-4d3b-b671-49534e93b477\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") "
Sep 08 10:27:18.105434 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.105216    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-dshm\") pod \"3ef733af-b8be-4d3b-b671-49534e93b477\" (UID: \"3ef733af-b8be-4d3b-b671-49534e93b477\") "
Sep 08 10:27:18.105434 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.105287    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-home" (OuterVolumeSpecName: "home") pod "3ef733af-b8be-4d3b-b671-49534e93b477" (UID: "3ef733af-b8be-4d3b-b671-49534e93b477"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:27:18.105541 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.105451    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ef733af-b8be-4d3b-b671-49534e93b477" (UID: "3ef733af-b8be-4d3b-b671-49534e93b477"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:27:18.105541 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.105505    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ef733af-b8be-4d3b-b671-49534e93b477" (UID: "3ef733af-b8be-4d3b-b671-49534e93b477"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:27:18.105646 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.105605    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-home\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:27:18.105646 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.105624    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-tmp-dir\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:27:18.105646 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.105639    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-model-cache\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:27:18.107321 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.107296    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-dshm" (OuterVolumeSpecName: "dshm") pod "3ef733af-b8be-4d3b-b671-49534e93b477" (UID: "3ef733af-b8be-4d3b-b671-49534e93b477"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:27:18.107426 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.107367    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef733af-b8be-4d3b-b671-49534e93b477-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ef733af-b8be-4d3b-b671-49534e93b477" (UID: "3ef733af-b8be-4d3b-b671-49534e93b477"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:27:18.107426 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.107380    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef733af-b8be-4d3b-b671-49534e93b477-kube-api-access-pzcpf" (OuterVolumeSpecName: "kube-api-access-pzcpf") pod "3ef733af-b8be-4d3b-b671-49534e93b477" (UID: "3ef733af-b8be-4d3b-b671-49534e93b477"). InnerVolumeSpecName "kube-api-access-pzcpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:27:18.206505 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.206483    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef733af-b8be-4d3b-b671-49534e93b477-dshm\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:27:18.206505 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.206504    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef733af-b8be-4d3b-b671-49534e93b477-tls-certs\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:27:18.206669 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.206515    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzcpf\" (UniqueName: \"kubernetes.io/projected/3ef733af-b8be-4d3b-b671-49534e93b477-kube-api-access-pzcpf\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:27:18.755829 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.755769    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp" podUID="f02bee9e-ed6f-4d12-9460-9aa36b42fe62" containerName="main" containerID="cri-o://51413d5a1e7d7f517e7b2f75e9ca3ed0c6b349fcf58dc58da8d6c9164a9097e2" gracePeriod=45
Sep 08 10:27:18.779021 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.778996    2559 generic.go:356] "Generic (PLEG): container finished" podID="3ef733af-b8be-4d3b-b671-49534e93b477" containerID="2bbb6ed29c69a56040e395de99198dcebcf05584990e03a24bbb2d2f74a7643f" exitCode=0
Sep 08 10:27:18.779352 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.779039    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"
Sep 08 10:27:18.779352 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.779056    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h" event={"ID":"3ef733af-b8be-4d3b-b671-49534e93b477","Type":"ContainerDied","Data":"2bbb6ed29c69a56040e395de99198dcebcf05584990e03a24bbb2d2f74a7643f"}
Sep 08 10:27:18.779352 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.779100    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h" event={"ID":"3ef733af-b8be-4d3b-b671-49534e93b477","Type":"ContainerDied","Data":"d8115cd16de9d952b0b5ce10181b71748c613dbb8b96dab8baea2a341d3cce82"}
Sep 08 10:27:18.779352 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.779124    2559 scope.go:117] "RemoveContainer" containerID="2bbb6ed29c69a56040e395de99198dcebcf05584990e03a24bbb2d2f74a7643f"
Sep 08 10:27:18.832850 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.832827    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"]
Sep 08 10:27:18.836967 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.836947    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-71a7a1/rate-test-model-71a7a1-kserve-647ccfd99-h589h"]
Sep 08 10:27:18.866567 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.866543    2559 scope.go:117] "RemoveContainer" containerID="2bbb6ed29c69a56040e395de99198dcebcf05584990e03a24bbb2d2f74a7643f"
Sep 08 10:27:18.866853 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:27:18.866833    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bbb6ed29c69a56040e395de99198dcebcf05584990e03a24bbb2d2f74a7643f\": container with ID starting with 2bbb6ed29c69a56040e395de99198dcebcf05584990e03a24bbb2d2f74a7643f not found: ID does not exist" containerID="2bbb6ed29c69a56040e395de99198dcebcf05584990e03a24bbb2d2f74a7643f"
Sep 08 10:27:18.866907 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:18.866862    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbb6ed29c69a56040e395de99198dcebcf05584990e03a24bbb2d2f74a7643f"} err="failed to get container status \"2bbb6ed29c69a56040e395de99198dcebcf05584990e03a24bbb2d2f74a7643f\": rpc error: code = NotFound desc = could not find container \"2bbb6ed29c69a56040e395de99198dcebcf05584990e03a24bbb2d2f74a7643f\": container with ID starting with 2bbb6ed29c69a56040e395de99198dcebcf05584990e03a24bbb2d2f74a7643f not found: ID does not exist"
Sep 08 10:27:19.012107 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.012086    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:27:19.114746 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.114716    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-dshm\") pod \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") "
Sep 08 10:27:19.114746 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.114750    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-tls-certs\") pod \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") "
Sep 08 10:27:19.114978 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.114803    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-home\") pod \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") "
Sep 08 10:27:19.114978 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.114842    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-tmp-dir\") pod \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") "
Sep 08 10:27:19.114978 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.114903    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-model-cache\") pod \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") "
Sep 08 10:27:19.114978 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.114920    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnxm5\" (UniqueName: \"kubernetes.io/projected/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-kube-api-access-gnxm5\") pod \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\" (UID: \"f02bee9e-ed6f-4d12-9460-9aa36b42fe62\") "
Sep 08 10:27:19.115180 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.115125    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-home" (OuterVolumeSpecName: "home") pod "f02bee9e-ed6f-4d12-9460-9aa36b42fe62" (UID: "f02bee9e-ed6f-4d12-9460-9aa36b42fe62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:27:19.115180 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.115164    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f02bee9e-ed6f-4d12-9460-9aa36b42fe62" (UID: "f02bee9e-ed6f-4d12-9460-9aa36b42fe62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:27:19.115301 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.115170    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-model-cache" (OuterVolumeSpecName: "model-cache") pod "f02bee9e-ed6f-4d12-9460-9aa36b42fe62" (UID: "f02bee9e-ed6f-4d12-9460-9aa36b42fe62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:27:19.116923 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.116898    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-dshm" (OuterVolumeSpecName: "dshm") pod "f02bee9e-ed6f-4d12-9460-9aa36b42fe62" (UID: "f02bee9e-ed6f-4d12-9460-9aa36b42fe62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:27:19.117074 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.117051    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f02bee9e-ed6f-4d12-9460-9aa36b42fe62" (UID: "f02bee9e-ed6f-4d12-9460-9aa36b42fe62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:27:19.117175 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.117099    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-kube-api-access-gnxm5" (OuterVolumeSpecName: "kube-api-access-gnxm5") pod "f02bee9e-ed6f-4d12-9460-9aa36b42fe62" (UID: "f02bee9e-ed6f-4d12-9460-9aa36b42fe62"). InnerVolumeSpecName "kube-api-access-gnxm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:27:19.216587 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.216531    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-model-cache\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:27:19.216587 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.216557    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnxm5\" (UniqueName: \"kubernetes.io/projected/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-kube-api-access-gnxm5\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:27:19.216587 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.216567    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-dshm\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:27:19.216587 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.216576    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-tls-certs\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:27:19.216587 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.216585    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-home\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:27:19.216807 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.216609    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02bee9e-ed6f-4d12-9460-9aa36b42fe62-tmp-dir\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:27:19.785743 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.785715    2559 generic.go:356] "Generic (PLEG): container finished" podID="f02bee9e-ed6f-4d12-9460-9aa36b42fe62" containerID="51413d5a1e7d7f517e7b2f75e9ca3ed0c6b349fcf58dc58da8d6c9164a9097e2" exitCode=0
Sep 08 10:27:19.786209 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.785769    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"
Sep 08 10:27:19.786209 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.785798    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp" event={"ID":"f02bee9e-ed6f-4d12-9460-9aa36b42fe62","Type":"ContainerDied","Data":"51413d5a1e7d7f517e7b2f75e9ca3ed0c6b349fcf58dc58da8d6c9164a9097e2"}
Sep 08 10:27:19.786209 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.785847    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp" event={"ID":"f02bee9e-ed6f-4d12-9460-9aa36b42fe62","Type":"ContainerDied","Data":"97ca086a9c4d0fe453cd7046848bf8cd0b492667c059408def9b0e2478999a0c"}
Sep 08 10:27:19.786209 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.785872    2559 scope.go:117] "RemoveContainer" containerID="51413d5a1e7d7f517e7b2f75e9ca3ed0c6b349fcf58dc58da8d6c9164a9097e2"
Sep 08 10:27:19.794534 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.794518    2559 scope.go:117] "RemoveContainer" containerID="51413d5a1e7d7f517e7b2f75e9ca3ed0c6b349fcf58dc58da8d6c9164a9097e2"
Sep 08 10:27:19.794757 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:27:19.794742    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51413d5a1e7d7f517e7b2f75e9ca3ed0c6b349fcf58dc58da8d6c9164a9097e2\": container with ID starting with 51413d5a1e7d7f517e7b2f75e9ca3ed0c6b349fcf58dc58da8d6c9164a9097e2 not found: ID does not exist" containerID="51413d5a1e7d7f517e7b2f75e9ca3ed0c6b349fcf58dc58da8d6c9164a9097e2"
Sep 08 10:27:19.794811 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.794764    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51413d5a1e7d7f517e7b2f75e9ca3ed0c6b349fcf58dc58da8d6c9164a9097e2"} err="failed to get container status \"51413d5a1e7d7f517e7b2f75e9ca3ed0c6b349fcf58dc58da8d6c9164a9097e2\": rpc error: code = NotFound desc = could not find container \"51413d5a1e7d7f517e7b2f75e9ca3ed0c6b349fcf58dc58da8d6c9164a9097e2\": container with ID starting with 51413d5a1e7d7f517e7b2f75e9ca3ed0c6b349fcf58dc58da8d6c9164a9097e2 not found: ID does not exist"
Sep 08 10:27:19.807207 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.807184    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"]
Sep 08 10:27:19.814090 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:19.814070    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d6e53c/rate-test-model-d6e53c-kserve-88b87b4dc-mq5kp"]
Sep 08 10:27:20.636419 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:20.636390    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ef733af-b8be-4d3b-b671-49534e93b477" path="/var/lib/kubelet/pods/3ef733af-b8be-4d3b-b671-49534e93b477/volumes"
Sep 08 10:27:20.636793 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:20.636779    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f02bee9e-ed6f-4d12-9460-9aa36b42fe62" path="/var/lib/kubelet/pods/f02bee9e-ed6f-4d12-9460-9aa36b42fe62/volumes"
Sep 08 10:27:20.971914 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:20.971871    2559 scope.go:117] "RemoveContainer" containerID="acc10ba5490fa1595f698e5e6dd15a2ad237eb14e97bc7df552954e222ac0e50"
Sep 08 10:27:38.576116 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.576083    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"]
Sep 08 10:27:38.576605 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.576482    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ef733af-b8be-4d3b-b671-49534e93b477" containerName="main"
Sep 08 10:27:38.576605 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.576494    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ef733af-b8be-4d3b-b671-49534e93b477" containerName="main"
Sep 08 10:27:38.576605 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.576505    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8e0fa7-f53d-400e-abcf-2b9f95fc0892" containerName="payload-processing"
Sep 08 10:27:38.576605 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.576511    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8e0fa7-f53d-400e-abcf-2b9f95fc0892" containerName="payload-processing"
Sep 08 10:27:38.576605 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.576534    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02bee9e-ed6f-4d12-9460-9aa36b42fe62" containerName="main"
Sep 08 10:27:38.576605 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.576543    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02bee9e-ed6f-4d12-9460-9aa36b42fe62" containerName="main"
Sep 08 10:27:38.576605 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.576604    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d8e0fa7-f53d-400e-abcf-2b9f95fc0892" containerName="payload-processing"
Sep 08 10:27:38.576605 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.576613    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ef733af-b8be-4d3b-b671-49534e93b477" containerName="main"
Sep 08 10:27:38.576957 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.576621    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f02bee9e-ed6f-4d12-9460-9aa36b42fe62" containerName="main"
Sep 08 10:27:38.579785 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.579768    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"
Sep 08 10:27:38.582281 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.582258    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-86902f\""
Sep 08 10:27:38.583185 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.583165    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-86902f-dockercfg-ppdpd\""
Sep 08 10:27:38.592511 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.592488    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"]
Sep 08 10:27:38.594355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.594335    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"]
Sep 08 10:27:38.597920 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.597895    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"
Sep 08 10:27:38.608325 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.608236    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"]
Sep 08 10:27:38.678216 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.678191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq77g\" (UniqueName: \"kubernetes.io/projected/0273a177-f332-47df-b8f3-8384fab5754d-kube-api-access-nq77g\") pod \"payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2\" (UID: \"0273a177-f332-47df-b8f3-8384fab5754d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"
Sep 08 10:27:38.678355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.678224    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0273a177-f332-47df-b8f3-8384fab5754d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2\" (UID: \"0273a177-f332-47df-b8f3-8384fab5754d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"
Sep 08 10:27:38.678355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.678270    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a571f371-fe5b-4fc1-8abb-483a5f5cc72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx\" (UID: \"a571f371-fe5b-4fc1-8abb-483a5f5cc72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"
Sep 08 10:27:38.678355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.678319    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88s84\" (UniqueName: \"kubernetes.io/projected/a571f371-fe5b-4fc1-8abb-483a5f5cc72c-kube-api-access-88s84\") pod \"payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx\" (UID: \"a571f371-fe5b-4fc1-8abb-483a5f5cc72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"
Sep 08 10:27:38.778990 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.778961    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq77g\" (UniqueName: \"kubernetes.io/projected/0273a177-f332-47df-b8f3-8384fab5754d-kube-api-access-nq77g\") pod \"payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2\" (UID: \"0273a177-f332-47df-b8f3-8384fab5754d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"
Sep 08 10:27:38.779115 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.778993    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0273a177-f332-47df-b8f3-8384fab5754d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2\" (UID: \"0273a177-f332-47df-b8f3-8384fab5754d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"
Sep 08 10:27:38.779115 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.779014    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a571f371-fe5b-4fc1-8abb-483a5f5cc72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx\" (UID: \"a571f371-fe5b-4fc1-8abb-483a5f5cc72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"
Sep 08 10:27:38.779115 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.779043    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88s84\" (UniqueName: \"kubernetes.io/projected/a571f371-fe5b-4fc1-8abb-483a5f5cc72c-kube-api-access-88s84\") pod \"payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx\" (UID: \"a571f371-fe5b-4fc1-8abb-483a5f5cc72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"
Sep 08 10:27:38.779714 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.779696    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0273a177-f332-47df-b8f3-8384fab5754d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2\" (UID: \"0273a177-f332-47df-b8f3-8384fab5754d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"
Sep 08 10:27:38.779769 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.779748    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a571f371-fe5b-4fc1-8abb-483a5f5cc72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx\" (UID: \"a571f371-fe5b-4fc1-8abb-483a5f5cc72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"
Sep 08 10:27:38.787806 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.787785    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88s84\" (UniqueName: \"kubernetes.io/projected/a571f371-fe5b-4fc1-8abb-483a5f5cc72c-kube-api-access-88s84\") pod \"payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx\" (UID: \"a571f371-fe5b-4fc1-8abb-483a5f5cc72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"
Sep 08 10:27:38.788132 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.788111    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq77g\" (UniqueName: \"kubernetes.io/projected/0273a177-f332-47df-b8f3-8384fab5754d-kube-api-access-nq77g\") pod \"payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2\" (UID: \"0273a177-f332-47df-b8f3-8384fab5754d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"
Sep 08 10:27:38.889737 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.889673    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"
Sep 08 10:27:38.909360 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:38.909338    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"
Sep 08 10:27:39.166750 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:39.166724    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"]
Sep 08 10:27:39.168819 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:27:39.168793    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda571f371_fe5b_4fc1_8abb_483a5f5cc72c.slice/crio-8d584945b170520af82b1706b37cf60083e1c092d59a1091bec7b7bd69961283 WatchSource:0}: Error finding container 8d584945b170520af82b1706b37cf60083e1c092d59a1091bec7b7bd69961283: Status 404 returned error can't find the container with id 8d584945b170520af82b1706b37cf60083e1c092d59a1091bec7b7bd69961283
Sep 08 10:27:39.191272 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:39.191229    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"]
Sep 08 10:27:39.201390 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:27:39.201362    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0273a177_f332_47df_b8f3_8384fab5754d.slice/crio-d8e6be35c4f0a8d133f3855900479555f9b71c5c6d4af7d7cf38cdbc70547f37 WatchSource:0}: Error finding container d8e6be35c4f0a8d133f3855900479555f9b71c5c6d4af7d7cf38cdbc70547f37: Status 404 returned error can't find the container with id d8e6be35c4f0a8d133f3855900479555f9b71c5c6d4af7d7cf38cdbc70547f37
Sep 08 10:27:39.893564 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:39.893530    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2" event={"ID":"0273a177-f332-47df-b8f3-8384fab5754d","Type":"ContainerStarted","Data":"db69e8890561cf7eda7b1c99f0815418dae8851c3de6127075ca653fa0664b7b"}
Sep 08 10:27:39.894039 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:39.893582    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2" event={"ID":"0273a177-f332-47df-b8f3-8384fab5754d","Type":"ContainerStarted","Data":"d8e6be35c4f0a8d133f3855900479555f9b71c5c6d4af7d7cf38cdbc70547f37"}
Sep 08 10:27:39.894039 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:39.893704    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"
Sep 08 10:27:39.895291 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:39.895267    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx" event={"ID":"a571f371-fe5b-4fc1-8abb-483a5f5cc72c","Type":"ContainerStarted","Data":"1856dd4a5f67dbdd1e72d1a4121b8793dfb0282b55e517fbcf5135d5cb3b3640"}
Sep 08 10:27:39.895388 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:39.895304    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx" event={"ID":"a571f371-fe5b-4fc1-8abb-483a5f5cc72c","Type":"ContainerStarted","Data":"8d584945b170520af82b1706b37cf60083e1c092d59a1091bec7b7bd69961283"}
Sep 08 10:27:39.895388 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:39.895368    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"
Sep 08 10:27:39.912643 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:39.912592    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2" podStartSLOduration=1.912579003 podStartE2EDuration="1.912579003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:27:39.91016694 +0000 UTC m=+1579.751391342" watchObservedRunningTime="2026-09-08 10:27:39.912579003 +0000 UTC m=+1579.753803385"
Sep 08 10:27:39.926819 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:39.926679    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx" podStartSLOduration=1.926661892 podStartE2EDuration="1.926661892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:27:39.926151322 +0000 UTC m=+1579.767375729" watchObservedRunningTime="2026-09-08 10:27:39.926661892 +0000 UTC m=+1579.767886276"
Sep 08 10:27:50.903015 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:50.902984    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"
Sep 08 10:27:50.903395 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:27:50.903051    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"
Sep 08 10:28:06.845911 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:06.845868    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7ld5x_31528300-0a49-409d-b0c4-f3083656cc48/payload-processing/0.log"
Sep 08 10:28:06.965633 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:06.965608    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2_0273a177-f332-47df-b8f3-8384fab5754d/payload-processing/0.log"
Sep 08 10:28:20.999893 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:20.999829    2559 scope.go:117] "RemoveContainer" containerID="a75868019a74463619450d7af8144496e5ae12f992ea00d6b2255e4c03357bdc"
Sep 08 10:28:23.855149 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:23.855113    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"]
Sep 08 10:28:23.858694 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:23.858674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:23.862191 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:23.862156    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a0d0ba\"/\"ipp-route-a0d0ba-kserve-self-signed-certs\""
Sep 08 10:28:23.862191 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:23.862188    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a0d0ba\"/\"openshift-service-ca.crt\""
Sep 08 10:28:23.862374 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:23.862268    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a0d0ba\"/\"kube-root-ca.crt\""
Sep 08 10:28:23.862374 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:23.862160    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a0d0ba\"/\"default-dockercfg-ttc4x\""
Sep 08 10:28:23.869482 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:23.869460    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"]
Sep 08 10:28:23.966928 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:23.966904    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-model-cache\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:23.967054 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:23.966951    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-home\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:23.967054 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:23.967046    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-tls-certs\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:23.967129 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:23.967072    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-tmp-dir\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:23.967129 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:23.967093    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bkq6\" (UniqueName: \"kubernetes.io/projected/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-kube-api-access-5bkq6\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:23.967129 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:23.967117    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-dshm\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:24.067896 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:24.067872    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-tls-certs\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:24.068032 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:24.067901    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-tmp-dir\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:24.068032 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:24.067920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bkq6\" (UniqueName: \"kubernetes.io/projected/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-kube-api-access-5bkq6\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:24.068032 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:24.067937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-dshm\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:24.068238 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:24.068046    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-model-cache\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:24.068238 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:24.068126    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-home\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:24.068383 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:24.068341    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-tmp-dir\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:24.068443 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:24.068425    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-model-cache\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:24.068525 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:24.068470    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-home\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:24.070334 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:24.070305    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-dshm\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:24.070507 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:24.070489    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-tls-certs\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:24.077141 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:24.077122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bkq6\" (UniqueName: \"kubernetes.io/projected/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-kube-api-access-5bkq6\") pod \"ipp-route-a0d0ba-kserve-759c555b4c-zdn85\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") " pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:24.169666 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:24.169643    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:24.451475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:24.451448    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"]
Sep 08 10:28:24.452594 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:28:24.452565    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc3b55a2_0ea3_4933_b7fc_48f182c9af58.slice/crio-63f7e2ea0105ea2fa9cc245ac578e41bb2286cd8eaeee680ee20bdd80535ea02 WatchSource:0}: Error finding container 63f7e2ea0105ea2fa9cc245ac578e41bb2286cd8eaeee680ee20bdd80535ea02: Status 404 returned error can't find the container with id 63f7e2ea0105ea2fa9cc245ac578e41bb2286cd8eaeee680ee20bdd80535ea02
Sep 08 10:28:25.137280 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:25.137224    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85" event={"ID":"fc3b55a2-0ea3-4933-b7fc-48f182c9af58","Type":"ContainerStarted","Data":"f580ce477d6c6544a8cfc78ba268008b52c34daa5ee36e9f0ea4ca802e69e23e"}
Sep 08 10:28:25.137280 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:25.137282    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85" event={"ID":"fc3b55a2-0ea3-4933-b7fc-48f182c9af58","Type":"ContainerStarted","Data":"63f7e2ea0105ea2fa9cc245ac578e41bb2286cd8eaeee680ee20bdd80535ea02"}
Sep 08 10:28:25.181902 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:25.181851    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85" podStartSLOduration=2.181836343 podStartE2EDuration="2.181836343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:28:25.180498635 +0000 UTC m=+1625.021723037" watchObservedRunningTime="2026-09-08 10:28:25.181836343 +0000 UTC m=+1625.023060726"
Sep 08 10:28:34.170545 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:34.170505    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:34.170925 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:34.170561    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:34.184225 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:34.184203    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:34.195502 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:34.195482    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:28:55.111223 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:55.111183    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7ld5x_31528300-0a49-409d-b0c4-f3083656cc48/payload-processing/0.log"
Sep 08 10:28:55.229773 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:28:55.229753    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2_0273a177-f332-47df-b8f3-8384fab5754d/payload-processing/0.log"
Sep 08 10:29:00.492612 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:00.492580    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"]
Sep 08 10:29:00.496055 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:00.496038    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"
Sep 08 10:29:00.498667 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:00.498645    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-56bb3c-dockercfg-v5g52\""
Sep 08 10:29:00.498810 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:00.498748    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-56bb3c\""
Sep 08 10:29:00.505363 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:00.505338    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"]
Sep 08 10:29:00.574686 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:00.574651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbkt\" (UniqueName: \"kubernetes.io/projected/3dff330e-c547-480e-9aee-fa3699dfa20f-kube-api-access-8zbkt\") pod \"payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j\" (UID: \"3dff330e-c547-480e-9aee-fa3699dfa20f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"
Sep 08 10:29:00.574686 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:00.574688    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dff330e-c547-480e-9aee-fa3699dfa20f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j\" (UID: \"3dff330e-c547-480e-9aee-fa3699dfa20f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"
Sep 08 10:29:00.675887 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:00.675857    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbkt\" (UniqueName: \"kubernetes.io/projected/3dff330e-c547-480e-9aee-fa3699dfa20f-kube-api-access-8zbkt\") pod \"payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j\" (UID: \"3dff330e-c547-480e-9aee-fa3699dfa20f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"
Sep 08 10:29:00.675887 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:00.675894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dff330e-c547-480e-9aee-fa3699dfa20f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j\" (UID: \"3dff330e-c547-480e-9aee-fa3699dfa20f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"
Sep 08 10:29:00.676659 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:00.676640    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dff330e-c547-480e-9aee-fa3699dfa20f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j\" (UID: \"3dff330e-c547-480e-9aee-fa3699dfa20f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"
Sep 08 10:29:00.692296 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:00.692256    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbkt\" (UniqueName: \"kubernetes.io/projected/3dff330e-c547-480e-9aee-fa3699dfa20f-kube-api-access-8zbkt\") pod \"payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j\" (UID: \"3dff330e-c547-480e-9aee-fa3699dfa20f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"
Sep 08 10:29:00.809187 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:00.809103    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"
Sep 08 10:29:01.084748 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:01.084641    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"]
Sep 08 10:29:01.087269 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:29:01.087221    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dff330e_c547_480e_9aee_fa3699dfa20f.slice/crio-c0377716c7bb00a92881815812ce7e138e875f5323dcabdf86c620c08bfcea9f WatchSource:0}: Error finding container c0377716c7bb00a92881815812ce7e138e875f5323dcabdf86c620c08bfcea9f: Status 404 returned error can't find the container with id c0377716c7bb00a92881815812ce7e138e875f5323dcabdf86c620c08bfcea9f
Sep 08 10:29:01.327873 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:01.327841    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j" event={"ID":"3dff330e-c547-480e-9aee-fa3699dfa20f","Type":"ContainerStarted","Data":"aa65bd529aceea2042a5a0543c6eb1701f021d70331ddf0d48b98531b29d1a19"}
Sep 08 10:29:01.328039 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:01.327893    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j" event={"ID":"3dff330e-c547-480e-9aee-fa3699dfa20f","Type":"ContainerStarted","Data":"c0377716c7bb00a92881815812ce7e138e875f5323dcabdf86c620c08bfcea9f"}
Sep 08 10:29:01.328039 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:01.328001    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"
Sep 08 10:29:12.335301 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:12.335235    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"
Sep 08 10:29:12.355088 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:12.355040    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j" podStartSLOduration=12.355026471 podStartE2EDuration="12.355026471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:29:01.368917832 +0000 UTC m=+1661.210142214" watchObservedRunningTime="2026-09-08 10:29:12.355026471 +0000 UTC m=+1672.196250888"
Sep 08 10:29:24.188634 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:24.188599    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq"]
Sep 08 10:29:24.192189 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:24.192168    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq"
Sep 08 10:29:24.196512 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:24.196490    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:29:24.196732 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:24.196716    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:29:24.206009 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:24.205989    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq"]
Sep 08 10:29:24.290022 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:24.289995    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a06c91-fc97-4d81-ac4b-a9b1bd3a430e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq\" (UID: \"28a06c91-fc97-4d81-ac4b-a9b1bd3a430e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq"
Sep 08 10:29:24.391085 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:24.391059    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a06c91-fc97-4d81-ac4b-a9b1bd3a430e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq\" (UID: \"28a06c91-fc97-4d81-ac4b-a9b1bd3a430e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq"
Sep 08 10:29:24.391682 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:24.391661    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a06c91-fc97-4d81-ac4b-a9b1bd3a430e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq\" (UID: \"28a06c91-fc97-4d81-ac4b-a9b1bd3a430e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq"
Sep 08 10:29:24.502862 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:24.502806    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq"
Sep 08 10:29:24.780427 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:24.780404    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq"]
Sep 08 10:29:24.782415 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:29:24.782372    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a06c91_fc97_4d81_ac4b_a9b1bd3a430e.slice/crio-f5d3e55247b794c292b018c47914d4fe3a9aa279f456d083a22759ae5f0dc01f WatchSource:0}: Error finding container f5d3e55247b794c292b018c47914d4fe3a9aa279f456d083a22759ae5f0dc01f: Status 404 returned error can't find the container with id f5d3e55247b794c292b018c47914d4fe3a9aa279f456d083a22759ae5f0dc01f
Sep 08 10:29:25.461517 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:25.461476    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq" event={"ID":"28a06c91-fc97-4d81-ac4b-a9b1bd3a430e","Type":"ContainerStarted","Data":"9c10518eb93d85cf69aef0813322916f8b4fd2ee7274cb058cb3133dd7efec2c"}
Sep 08 10:29:25.461902 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:25.461535    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq" event={"ID":"28a06c91-fc97-4d81-ac4b-a9b1bd3a430e","Type":"ContainerStarted","Data":"f5d3e55247b794c292b018c47914d4fe3a9aa279f456d083a22759ae5f0dc01f"}
Sep 08 10:29:25.477492 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:25.477446    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq" podStartSLOduration=1.477431244 podStartE2EDuration="1.477431244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:29:25.475684823 +0000 UTC m=+1685.316909206" watchObservedRunningTime="2026-09-08 10:29:25.477431244 +0000 UTC m=+1685.318655650"
Sep 08 10:29:26.468834 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:26.468799    2559 generic.go:356] "Generic (PLEG): container finished" podID="28a06c91-fc97-4d81-ac4b-a9b1bd3a430e" containerID="9c10518eb93d85cf69aef0813322916f8b4fd2ee7274cb058cb3133dd7efec2c" exitCode=0
Sep 08 10:29:26.469494 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:26.468886    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq" event={"ID":"28a06c91-fc97-4d81-ac4b-a9b1bd3a430e","Type":"ContainerDied","Data":"9c10518eb93d85cf69aef0813322916f8b4fd2ee7274cb058cb3133dd7efec2c"}
Sep 08 10:29:27.610079 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:27.610054    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq"
Sep 08 10:29:27.618765 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:27.618744    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a06c91-fc97-4d81-ac4b-a9b1bd3a430e-maas-api-service-ca\") pod \"28a06c91-fc97-4d81-ac4b-a9b1bd3a430e\" (UID: \"28a06c91-fc97-4d81-ac4b-a9b1bd3a430e\") "
Sep 08 10:29:27.619044 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:27.619023    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a06c91-fc97-4d81-ac4b-a9b1bd3a430e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "28a06c91-fc97-4d81-ac4b-a9b1bd3a430e" (UID: "28a06c91-fc97-4d81-ac4b-a9b1bd3a430e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:29:27.719590 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:27.719561    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a06c91-fc97-4d81-ac4b-a9b1bd3a430e-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:29:28.481540 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:28.481511    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq"
Sep 08 10:29:28.481540 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:28.481523    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq" event={"ID":"28a06c91-fc97-4d81-ac4b-a9b1bd3a430e","Type":"ContainerDied","Data":"f5d3e55247b794c292b018c47914d4fe3a9aa279f456d083a22759ae5f0dc01f"}
Sep 08 10:29:28.481802 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:28.481573    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5d3e55247b794c292b018c47914d4fe3a9aa279f456d083a22759ae5f0dc01f"
Sep 08 10:29:30.243882 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.243845    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"]
Sep 08 10:29:30.244356 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.244112    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j" podUID="3dff330e-c547-480e-9aee-fa3699dfa20f" containerName="payload-processing" containerID="cri-o://aa65bd529aceea2042a5a0543c6eb1701f021d70331ddf0d48b98531b29d1a19" gracePeriod=30
Sep 08 10:29:30.445799 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.445771    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq"]
Sep 08 10:29:30.454453 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.454431    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-56bb3c-rspdq"]
Sep 08 10:29:30.487237 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.487215    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"
Sep 08 10:29:30.493135 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.493112    2559 generic.go:356] "Generic (PLEG): container finished" podID="3dff330e-c547-480e-9aee-fa3699dfa20f" containerID="aa65bd529aceea2042a5a0543c6eb1701f021d70331ddf0d48b98531b29d1a19" exitCode=0
Sep 08 10:29:30.493226 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.493150    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"
Sep 08 10:29:30.493226 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.493163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j" event={"ID":"3dff330e-c547-480e-9aee-fa3699dfa20f","Type":"ContainerDied","Data":"aa65bd529aceea2042a5a0543c6eb1701f021d70331ddf0d48b98531b29d1a19"}
Sep 08 10:29:30.493226 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.493202    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j" event={"ID":"3dff330e-c547-480e-9aee-fa3699dfa20f","Type":"ContainerDied","Data":"c0377716c7bb00a92881815812ce7e138e875f5323dcabdf86c620c08bfcea9f"}
Sep 08 10:29:30.493226 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.493225    2559 scope.go:117] "RemoveContainer" containerID="aa65bd529aceea2042a5a0543c6eb1701f021d70331ddf0d48b98531b29d1a19"
Sep 08 10:29:30.502015 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.501999    2559 scope.go:117] "RemoveContainer" containerID="aa65bd529aceea2042a5a0543c6eb1701f021d70331ddf0d48b98531b29d1a19"
Sep 08 10:29:30.502323 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:29:30.502304    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa65bd529aceea2042a5a0543c6eb1701f021d70331ddf0d48b98531b29d1a19\": container with ID starting with aa65bd529aceea2042a5a0543c6eb1701f021d70331ddf0d48b98531b29d1a19 not found: ID does not exist" containerID="aa65bd529aceea2042a5a0543c6eb1701f021d70331ddf0d48b98531b29d1a19"
Sep 08 10:29:30.502388 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.502331    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa65bd529aceea2042a5a0543c6eb1701f021d70331ddf0d48b98531b29d1a19"} err="failed to get container status \"aa65bd529aceea2042a5a0543c6eb1701f021d70331ddf0d48b98531b29d1a19\": rpc error: code = NotFound desc = could not find container \"aa65bd529aceea2042a5a0543c6eb1701f021d70331ddf0d48b98531b29d1a19\": container with ID starting with aa65bd529aceea2042a5a0543c6eb1701f021d70331ddf0d48b98531b29d1a19 not found: ID does not exist"
Sep 08 10:29:30.543212 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.543191    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zbkt\" (UniqueName: \"kubernetes.io/projected/3dff330e-c547-480e-9aee-fa3699dfa20f-kube-api-access-8zbkt\") pod \"3dff330e-c547-480e-9aee-fa3699dfa20f\" (UID: \"3dff330e-c547-480e-9aee-fa3699dfa20f\") "
Sep 08 10:29:30.543369 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.543352    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dff330e-c547-480e-9aee-fa3699dfa20f-plugins-config-volume\") pod \"3dff330e-c547-480e-9aee-fa3699dfa20f\" (UID: \"3dff330e-c547-480e-9aee-fa3699dfa20f\") "
Sep 08 10:29:30.543694 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.543674    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dff330e-c547-480e-9aee-fa3699dfa20f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3dff330e-c547-480e-9aee-fa3699dfa20f" (UID: "3dff330e-c547-480e-9aee-fa3699dfa20f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:29:30.545187 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.545169    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dff330e-c547-480e-9aee-fa3699dfa20f-kube-api-access-8zbkt" (OuterVolumeSpecName: "kube-api-access-8zbkt") pod "3dff330e-c547-480e-9aee-fa3699dfa20f" (UID: "3dff330e-c547-480e-9aee-fa3699dfa20f"). InnerVolumeSpecName "kube-api-access-8zbkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:29:30.636077 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.636054    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28a06c91-fc97-4d81-ac4b-a9b1bd3a430e" path="/var/lib/kubelet/pods/28a06c91-fc97-4d81-ac4b-a9b1bd3a430e/volumes"
Sep 08 10:29:30.644125 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.644105    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dff330e-c547-480e-9aee-fa3699dfa20f-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:29:30.644201 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.644127    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zbkt\" (UniqueName: \"kubernetes.io/projected/3dff330e-c547-480e-9aee-fa3699dfa20f-kube-api-access-8zbkt\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:29:30.809570 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.809512    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"]
Sep 08 10:29:30.813523 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:30.813500    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-56bb3c-76fb44c69c-zh99j"]
Sep 08 10:29:32.636287 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:32.636226    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dff330e-c547-480e-9aee-fa3699dfa20f" path="/var/lib/kubelet/pods/3dff330e-c547-480e-9aee-fa3699dfa20f/volumes"
Sep 08 10:29:40.674006 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.673976    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8"]
Sep 08 10:29:40.674424 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.674399    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dff330e-c547-480e-9aee-fa3699dfa20f" containerName="payload-processing"
Sep 08 10:29:40.674424 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.674412    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dff330e-c547-480e-9aee-fa3699dfa20f" containerName="payload-processing"
Sep 08 10:29:40.674506 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.674430    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28a06c91-fc97-4d81-ac4b-a9b1bd3a430e" containerName="revoke-keys"
Sep 08 10:29:40.674506 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.674436    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="28a06c91-fc97-4d81-ac4b-a9b1bd3a430e" containerName="revoke-keys"
Sep 08 10:29:40.674578 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.674520    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dff330e-c547-480e-9aee-fa3699dfa20f" containerName="payload-processing"
Sep 08 10:29:40.674578 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.674528    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="28a06c91-fc97-4d81-ac4b-a9b1bd3a430e" containerName="revoke-keys"
Sep 08 10:29:40.679065 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.679047    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8"
Sep 08 10:29:40.684212 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.684192    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:29:40.684365 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.684297    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:29:40.695162 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.695140    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8"]
Sep 08 10:29:40.729825 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.729796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204a2809-7aec-46df-9fb9-f52ad3a58fab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8\" (UID: \"204a2809-7aec-46df-9fb9-f52ad3a58fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8"
Sep 08 10:29:40.830776 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.830750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204a2809-7aec-46df-9fb9-f52ad3a58fab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8\" (UID: \"204a2809-7aec-46df-9fb9-f52ad3a58fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8"
Sep 08 10:29:40.831360 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.831341    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204a2809-7aec-46df-9fb9-f52ad3a58fab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8\" (UID: \"204a2809-7aec-46df-9fb9-f52ad3a58fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8"
Sep 08 10:29:40.988903 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:40.988841    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8"
Sep 08 10:29:41.259594 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:41.259570    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8"]
Sep 08 10:29:41.261756 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:29:41.261729    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204a2809_7aec_46df_9fb9_f52ad3a58fab.slice/crio-8953fe0710a7f4310449f81eac3739897ada35cd92011934fa72204f0e99d828 WatchSource:0}: Error finding container 8953fe0710a7f4310449f81eac3739897ada35cd92011934fa72204f0e99d828: Status 404 returned error can't find the container with id 8953fe0710a7f4310449f81eac3739897ada35cd92011934fa72204f0e99d828
Sep 08 10:29:41.558309 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:41.558284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8" event={"ID":"204a2809-7aec-46df-9fb9-f52ad3a58fab","Type":"ContainerStarted","Data":"8953fe0710a7f4310449f81eac3739897ada35cd92011934fa72204f0e99d828"}
Sep 08 10:29:42.564929 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:42.564892    2559 generic.go:356] "Generic (PLEG): container finished" podID="204a2809-7aec-46df-9fb9-f52ad3a58fab" containerID="9154b460b9461a9fa15b027e25b47a650504a3f1cf63da399ced6cd0675170fc" exitCode=0
Sep 08 10:29:42.565367 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:42.564978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8" event={"ID":"204a2809-7aec-46df-9fb9-f52ad3a58fab","Type":"ContainerDied","Data":"9154b460b9461a9fa15b027e25b47a650504a3f1cf63da399ced6cd0675170fc"}
Sep 08 10:29:43.705815 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:43.705793    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8"
Sep 08 10:29:43.754151 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:43.754122    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204a2809-7aec-46df-9fb9-f52ad3a58fab-maas-api-service-ca\") pod \"204a2809-7aec-46df-9fb9-f52ad3a58fab\" (UID: \"204a2809-7aec-46df-9fb9-f52ad3a58fab\") "
Sep 08 10:29:43.754488 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:43.754465    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204a2809-7aec-46df-9fb9-f52ad3a58fab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "204a2809-7aec-46df-9fb9-f52ad3a58fab" (UID: "204a2809-7aec-46df-9fb9-f52ad3a58fab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:29:43.855681 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:43.855618    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204a2809-7aec-46df-9fb9-f52ad3a58fab-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:29:44.578185 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:44.578152    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8" event={"ID":"204a2809-7aec-46df-9fb9-f52ad3a58fab","Type":"ContainerDied","Data":"8953fe0710a7f4310449f81eac3739897ada35cd92011934fa72204f0e99d828"}
Sep 08 10:29:44.578185 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:44.578180    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8"
Sep 08 10:29:44.578406 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:44.578200    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8953fe0710a7f4310449f81eac3739897ada35cd92011934fa72204f0e99d828"
Sep 08 10:29:56.360215 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:56.360181    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8"]
Sep 08 10:29:56.362068 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:56.362045    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a0d0ba-t5nw8"]
Sep 08 10:29:56.637456 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:29:56.637374    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="204a2809-7aec-46df-9fb9-f52ad3a58fab" path="/var/lib/kubelet/pods/204a2809-7aec-46df-9fb9-f52ad3a58fab/volumes"
Sep 08 10:30:00.141954 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.141918    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814390-9btzm"]
Sep 08 10:30:00.142510 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.142368    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="204a2809-7aec-46df-9fb9-f52ad3a58fab" containerName="revoke-keys"
Sep 08 10:30:00.142510 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.142381    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="204a2809-7aec-46df-9fb9-f52ad3a58fab" containerName="revoke-keys"
Sep 08 10:30:00.142510 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.142457    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="204a2809-7aec-46df-9fb9-f52ad3a58fab" containerName="revoke-keys"
Sep 08 10:30:00.145607 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.145586    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-71a7a1-29814390-w2p88"]
Sep 08 10:30:00.145755 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.145712    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814390-9btzm"
Sep 08 10:30:00.148428 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.148408    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:30:00.149148 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.149128    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-71a7a1-29814390-w2p88"
Sep 08 10:30:00.151135 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.151113    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814390-9btzm"]
Sep 08 10:30:00.155469 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.155452    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-71a7a1-29814390-w2p88"]
Sep 08 10:30:00.160206 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.160189    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814390-9btzm"
Sep 08 10:30:00.168609 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.168590    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-71a7a1-29814390-w2p88"
Sep 08 10:30:00.438811 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.438781    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814390-9btzm"]
Sep 08 10:30:00.443000 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:30:00.442974    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21f91ac_d00c_496e_b9ac_eb5411386599.slice/crio-d1a72a2c44595dd5da9053aee714a9aad977c649e7771b4f81d5ea8dede8dc95 WatchSource:0}: Error finding container d1a72a2c44595dd5da9053aee714a9aad977c649e7771b4f81d5ea8dede8dc95: Status 404 returned error can't find the container with id d1a72a2c44595dd5da9053aee714a9aad977c649e7771b4f81d5ea8dede8dc95
Sep 08 10:30:00.445825 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.445805    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-71a7a1-29814390-w2p88"]
Sep 08 10:30:00.446816 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:30:00.446794    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f2de54_024a_4a27_8b4c_3fe4108e781c.slice/crio-786b5d704c06b62f0251241256a4440bd02bb31e7b9a29d651caac2b675a4794 WatchSource:0}: Error finding container 786b5d704c06b62f0251241256a4440bd02bb31e7b9a29d651caac2b675a4794: Status 404 returned error can't find the container with id 786b5d704c06b62f0251241256a4440bd02bb31e7b9a29d651caac2b675a4794
Sep 08 10:30:00.664686 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.664657    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814390-9btzm" event={"ID":"f21f91ac-d00c-496e-b9ac-eb5411386599","Type":"ContainerStarted","Data":"d1a72a2c44595dd5da9053aee714a9aad977c649e7771b4f81d5ea8dede8dc95"}
Sep 08 10:30:00.665810 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:00.665787    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-71a7a1-29814390-w2p88" event={"ID":"e4f2de54-024a-4a27-8b4c-3fe4108e781c","Type":"ContainerStarted","Data":"786b5d704c06b62f0251241256a4440bd02bb31e7b9a29d651caac2b675a4794"}
Sep 08 10:30:01.673097 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:01.673065    2559 generic.go:356] "Generic (PLEG): container finished" podID="f21f91ac-d00c-496e-b9ac-eb5411386599" containerID="c7405c3e2e38586f1762f00db2a4df03ab5cd2b319dee538fa8e5e9d80f718cc" exitCode=0
Sep 08 10:30:01.673500 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:01.673151    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814390-9btzm" event={"ID":"f21f91ac-d00c-496e-b9ac-eb5411386599","Type":"ContainerDied","Data":"c7405c3e2e38586f1762f00db2a4df03ab5cd2b319dee538fa8e5e9d80f718cc"}
Sep 08 10:30:01.674628 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:01.674609    2559 generic.go:356] "Generic (PLEG): container finished" podID="e4f2de54-024a-4a27-8b4c-3fe4108e781c" containerID="9a6910dd21cf2ba243551afa601c364fd46e847a9bbf283bdc10fe224747a07a" exitCode=0
Sep 08 10:30:01.674724 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:01.674682    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-71a7a1-29814390-w2p88" event={"ID":"e4f2de54-024a-4a27-8b4c-3fe4108e781c","Type":"ContainerDied","Data":"9a6910dd21cf2ba243551afa601c364fd46e847a9bbf283bdc10fe224747a07a"}
Sep 08 10:30:01.728088 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:01.728060    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"]
Sep 08 10:30:02.826490 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:02.826466    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814390-9btzm"
Sep 08 10:30:02.829335 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:02.829315    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-71a7a1-29814390-w2p88"
Sep 08 10:30:03.686363 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:03.686329    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814390-9btzm"
Sep 08 10:30:03.686566 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:03.686372    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814390-9btzm" event={"ID":"f21f91ac-d00c-496e-b9ac-eb5411386599","Type":"ContainerDied","Data":"d1a72a2c44595dd5da9053aee714a9aad977c649e7771b4f81d5ea8dede8dc95"}
Sep 08 10:30:03.686566 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:03.686422    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1a72a2c44595dd5da9053aee714a9aad977c649e7771b4f81d5ea8dede8dc95"
Sep 08 10:30:03.687621 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:03.687601    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-71a7a1-29814390-w2p88"
Sep 08 10:30:03.687749 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:03.687601    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-71a7a1-29814390-w2p88" event={"ID":"e4f2de54-024a-4a27-8b4c-3fe4108e781c","Type":"ContainerDied","Data":"786b5d704c06b62f0251241256a4440bd02bb31e7b9a29d651caac2b675a4794"}
Sep 08 10:30:03.687749 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:03.687728    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="786b5d704c06b62f0251241256a4440bd02bb31e7b9a29d651caac2b675a4794"
Sep 08 10:30:16.747981 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:16.747937    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85" podUID="fc3b55a2-0ea3-4933-b7fc-48f182c9af58" containerName="main" containerID="cri-o://f580ce477d6c6544a8cfc78ba268008b52c34daa5ee36e9f0ea4ca802e69e23e" gracePeriod=45
Sep 08 10:30:16.992130 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:16.992107    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:30:17.042403 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.042333    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-model-cache\") pod \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") "
Sep 08 10:30:17.042561 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.042428    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-tls-certs\") pod \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") "
Sep 08 10:30:17.042561 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.042470    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-tmp-dir\") pod \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") "
Sep 08 10:30:17.042561 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.042521    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bkq6\" (UniqueName: \"kubernetes.io/projected/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-kube-api-access-5bkq6\") pod \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") "
Sep 08 10:30:17.042561 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.042558    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-dshm\") pod \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") "
Sep 08 10:30:17.042799 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.042587    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-home\") pod \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\" (UID: \"fc3b55a2-0ea3-4933-b7fc-48f182c9af58\") "
Sep 08 10:30:17.042860 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.042810    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc3b55a2-0ea3-4933-b7fc-48f182c9af58" (UID: "fc3b55a2-0ea3-4933-b7fc-48f182c9af58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:30:17.042915 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.042630    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc3b55a2-0ea3-4933-b7fc-48f182c9af58" (UID: "fc3b55a2-0ea3-4933-b7fc-48f182c9af58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:30:17.043089 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.043061    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-home" (OuterVolumeSpecName: "home") pod "fc3b55a2-0ea3-4933-b7fc-48f182c9af58" (UID: "fc3b55a2-0ea3-4933-b7fc-48f182c9af58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:30:17.044934 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.044903    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-dshm" (OuterVolumeSpecName: "dshm") pod "fc3b55a2-0ea3-4933-b7fc-48f182c9af58" (UID: "fc3b55a2-0ea3-4933-b7fc-48f182c9af58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:30:17.045260 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.045218    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-kube-api-access-5bkq6" (OuterVolumeSpecName: "kube-api-access-5bkq6") pod "fc3b55a2-0ea3-4933-b7fc-48f182c9af58" (UID: "fc3b55a2-0ea3-4933-b7fc-48f182c9af58"). InnerVolumeSpecName "kube-api-access-5bkq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:30:17.045260 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.045218    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc3b55a2-0ea3-4933-b7fc-48f182c9af58" (UID: "fc3b55a2-0ea3-4933-b7fc-48f182c9af58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:30:17.143471 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.143441    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-tls-certs\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:30:17.143471 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.143467    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-tmp-dir\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:30:17.143471 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.143477    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bkq6\" (UniqueName: \"kubernetes.io/projected/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-kube-api-access-5bkq6\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:30:17.143674 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.143487    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-dshm\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:30:17.143674 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.143496    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-home\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:30:17.143674 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.143504    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3b55a2-0ea3-4933-b7fc-48f182c9af58-model-cache\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:30:17.765156 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.765126    2559 generic.go:356] "Generic (PLEG): container finished" podID="fc3b55a2-0ea3-4933-b7fc-48f182c9af58" containerID="f580ce477d6c6544a8cfc78ba268008b52c34daa5ee36e9f0ea4ca802e69e23e" exitCode=0
Sep 08 10:30:17.765560 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.765165    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"
Sep 08 10:30:17.765560 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.765202    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85" event={"ID":"fc3b55a2-0ea3-4933-b7fc-48f182c9af58","Type":"ContainerDied","Data":"f580ce477d6c6544a8cfc78ba268008b52c34daa5ee36e9f0ea4ca802e69e23e"}
Sep 08 10:30:17.765560 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.765269    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85" event={"ID":"fc3b55a2-0ea3-4933-b7fc-48f182c9af58","Type":"ContainerDied","Data":"63f7e2ea0105ea2fa9cc245ac578e41bb2286cd8eaeee680ee20bdd80535ea02"}
Sep 08 10:30:17.765560 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.765292    2559 scope.go:117] "RemoveContainer" containerID="f580ce477d6c6544a8cfc78ba268008b52c34daa5ee36e9f0ea4ca802e69e23e"
Sep 08 10:30:17.774737 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.774721    2559 scope.go:117] "RemoveContainer" containerID="f580ce477d6c6544a8cfc78ba268008b52c34daa5ee36e9f0ea4ca802e69e23e"
Sep 08 10:30:17.774998 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:30:17.774978    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f580ce477d6c6544a8cfc78ba268008b52c34daa5ee36e9f0ea4ca802e69e23e\": container with ID starting with f580ce477d6c6544a8cfc78ba268008b52c34daa5ee36e9f0ea4ca802e69e23e not found: ID does not exist" containerID="f580ce477d6c6544a8cfc78ba268008b52c34daa5ee36e9f0ea4ca802e69e23e"
Sep 08 10:30:17.775061 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.775011    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f580ce477d6c6544a8cfc78ba268008b52c34daa5ee36e9f0ea4ca802e69e23e"} err="failed to get container status \"f580ce477d6c6544a8cfc78ba268008b52c34daa5ee36e9f0ea4ca802e69e23e\": rpc error: code = NotFound desc = could not find container \"f580ce477d6c6544a8cfc78ba268008b52c34daa5ee36e9f0ea4ca802e69e23e\": container with ID starting with f580ce477d6c6544a8cfc78ba268008b52c34daa5ee36e9f0ea4ca802e69e23e not found: ID does not exist"
Sep 08 10:30:17.786029 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.786001    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"]
Sep 08 10:30:17.790848 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:17.790827    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a0d0ba/ipp-route-a0d0ba-kserve-759c555b4c-zdn85"]
Sep 08 10:30:18.637355 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:18.637321    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc3b55a2-0ea3-4933-b7fc-48f182c9af58" path="/var/lib/kubelet/pods/fc3b55a2-0ea3-4933-b7fc-48f182c9af58/volumes"
Sep 08 10:30:45.279739 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.279705    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm"]
Sep 08 10:30:45.280176 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.280093    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f21f91ac-d00c-496e-b9ac-eb5411386599" containerName="cleanup"
Sep 08 10:30:45.280176 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.280104    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f21f91ac-d00c-496e-b9ac-eb5411386599" containerName="cleanup"
Sep 08 10:30:45.280176 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.280125    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f2de54-024a-4a27-8b4c-3fe4108e781c" containerName="cleanup"
Sep 08 10:30:45.280176 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.280130    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f2de54-024a-4a27-8b4c-3fe4108e781c" containerName="cleanup"
Sep 08 10:30:45.280176 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.280142    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc3b55a2-0ea3-4933-b7fc-48f182c9af58" containerName="main"
Sep 08 10:30:45.280176 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.280149    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc3b55a2-0ea3-4933-b7fc-48f182c9af58" containerName="main"
Sep 08 10:30:45.280395 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.280205    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f21f91ac-d00c-496e-b9ac-eb5411386599" containerName="cleanup"
Sep 08 10:30:45.280395 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.280215    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc3b55a2-0ea3-4933-b7fc-48f182c9af58" containerName="main"
Sep 08 10:30:45.280395 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.280222    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4f2de54-024a-4a27-8b4c-3fe4108e781c" containerName="cleanup"
Sep 08 10:30:45.283627 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.283610    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm"
Sep 08 10:30:45.286253 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.286221    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:30:45.286346 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.286326    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:30:45.291788 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.291767    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm"]
Sep 08 10:30:45.369394 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.369357    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c382de46-b490-4955-83c7-b4848293d9fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm\" (UID: \"c382de46-b490-4955-83c7-b4848293d9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm"
Sep 08 10:30:45.470717 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.470684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c382de46-b490-4955-83c7-b4848293d9fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm\" (UID: \"c382de46-b490-4955-83c7-b4848293d9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm"
Sep 08 10:30:45.471304 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.471287    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c382de46-b490-4955-83c7-b4848293d9fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm\" (UID: \"c382de46-b490-4955-83c7-b4848293d9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm"
Sep 08 10:30:45.594911 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.594850    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm"
Sep 08 10:30:45.864870 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.864843    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm"]
Sep 08 10:30:45.866403 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:30:45.866376    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc382de46_b490_4955_83c7_b4848293d9fa.slice/crio-096f4a76afa9e8c19767bff65f6361764d3399723e50e154db872cbf28fcc291 WatchSource:0}: Error finding container 096f4a76afa9e8c19767bff65f6361764d3399723e50e154db872cbf28fcc291: Status 404 returned error can't find the container with id 096f4a76afa9e8c19767bff65f6361764d3399723e50e154db872cbf28fcc291
Sep 08 10:30:45.914752 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:45.914727    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm" event={"ID":"c382de46-b490-4955-83c7-b4848293d9fa","Type":"ContainerStarted","Data":"096f4a76afa9e8c19767bff65f6361764d3399723e50e154db872cbf28fcc291"}
Sep 08 10:30:46.921108 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:46.921072    2559 generic.go:356] "Generic (PLEG): container finished" podID="c382de46-b490-4955-83c7-b4848293d9fa" containerID="eeea4a12b198dbb86a9c8550829b855f2b820cf0f9aa847d0271c1f5a5782cc9" exitCode=0
Sep 08 10:30:46.921533 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:46.921168    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm" event={"ID":"c382de46-b490-4955-83c7-b4848293d9fa","Type":"ContainerDied","Data":"eeea4a12b198dbb86a9c8550829b855f2b820cf0f9aa847d0271c1f5a5782cc9"}
Sep 08 10:30:48.059801 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:48.059776    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm"
Sep 08 10:30:48.196515 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:48.196432    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c382de46-b490-4955-83c7-b4848293d9fa-maas-api-service-ca\") pod \"c382de46-b490-4955-83c7-b4848293d9fa\" (UID: \"c382de46-b490-4955-83c7-b4848293d9fa\") "
Sep 08 10:30:48.196830 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:48.196805    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c382de46-b490-4955-83c7-b4848293d9fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c382de46-b490-4955-83c7-b4848293d9fa" (UID: "c382de46-b490-4955-83c7-b4848293d9fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:30:48.297935 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:48.297910    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c382de46-b490-4955-83c7-b4848293d9fa-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:30:48.933475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:48.933443    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm" event={"ID":"c382de46-b490-4955-83c7-b4848293d9fa","Type":"ContainerDied","Data":"096f4a76afa9e8c19767bff65f6361764d3399723e50e154db872cbf28fcc291"}
Sep 08 10:30:48.933475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:48.933466    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm"
Sep 08 10:30:48.933671 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:48.933496    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="096f4a76afa9e8c19767bff65f6361764d3399723e50e154db872cbf28fcc291"
Sep 08 10:30:55.779933 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:55.779900    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"]
Sep 08 10:30:55.780430 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:55.780206    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2" podUID="0273a177-f332-47df-b8f3-8384fab5754d" containerName="payload-processing" containerID="cri-o://db69e8890561cf7eda7b1c99f0815418dae8851c3de6127075ca653fa0664b7b" gracePeriod=30
Sep 08 10:30:55.786900 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:55.786873    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"]
Sep 08 10:30:55.787123 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:55.787076    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx" podUID="a571f371-fe5b-4fc1-8abb-483a5f5cc72c" containerName="payload-pre-processing" containerID="cri-o://1856dd4a5f67dbdd1e72d1a4121b8793dfb0282b55e517fbcf5135d5cb3b3640" gracePeriod=30
Sep 08 10:30:55.946493 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:55.946459    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm"]
Sep 08 10:30:55.951516 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:55.951478    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86902f-njvdm"]
Sep 08 10:30:55.974682 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:55.974652    2559 generic.go:356] "Generic (PLEG): container finished" podID="0273a177-f332-47df-b8f3-8384fab5754d" containerID="db69e8890561cf7eda7b1c99f0815418dae8851c3de6127075ca653fa0664b7b" exitCode=0
Sep 08 10:30:55.974799 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:55.974729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2" event={"ID":"0273a177-f332-47df-b8f3-8384fab5754d","Type":"ContainerDied","Data":"db69e8890561cf7eda7b1c99f0815418dae8851c3de6127075ca653fa0664b7b"}
Sep 08 10:30:55.976525 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:55.976502    2559 generic.go:356] "Generic (PLEG): container finished" podID="a571f371-fe5b-4fc1-8abb-483a5f5cc72c" containerID="1856dd4a5f67dbdd1e72d1a4121b8793dfb0282b55e517fbcf5135d5cb3b3640" exitCode=0
Sep 08 10:30:55.976624 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:55.976562    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx" event={"ID":"a571f371-fe5b-4fc1-8abb-483a5f5cc72c","Type":"ContainerDied","Data":"1856dd4a5f67dbdd1e72d1a4121b8793dfb0282b55e517fbcf5135d5cb3b3640"}
Sep 08 10:30:56.073764 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.073736    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"
Sep 08 10:30:56.076580 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.076562    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"
Sep 08 10:30:56.160925 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.160896    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq77g\" (UniqueName: \"kubernetes.io/projected/0273a177-f332-47df-b8f3-8384fab5754d-kube-api-access-nq77g\") pod \"0273a177-f332-47df-b8f3-8384fab5754d\" (UID: \"0273a177-f332-47df-b8f3-8384fab5754d\") "
Sep 08 10:30:56.161054 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.160995    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0273a177-f332-47df-b8f3-8384fab5754d-plugins-config-volume\") pod \"0273a177-f332-47df-b8f3-8384fab5754d\" (UID: \"0273a177-f332-47df-b8f3-8384fab5754d\") "
Sep 08 10:30:56.161054 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.161028    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88s84\" (UniqueName: \"kubernetes.io/projected/a571f371-fe5b-4fc1-8abb-483a5f5cc72c-kube-api-access-88s84\") pod \"a571f371-fe5b-4fc1-8abb-483a5f5cc72c\" (UID: \"a571f371-fe5b-4fc1-8abb-483a5f5cc72c\") "
Sep 08 10:30:56.161134 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.161059    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a571f371-fe5b-4fc1-8abb-483a5f5cc72c-plugins-config-volume\") pod \"a571f371-fe5b-4fc1-8abb-483a5f5cc72c\" (UID: \"a571f371-fe5b-4fc1-8abb-483a5f5cc72c\") "
Sep 08 10:30:56.161437 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.161408    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0273a177-f332-47df-b8f3-8384fab5754d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0273a177-f332-47df-b8f3-8384fab5754d" (UID: "0273a177-f332-47df-b8f3-8384fab5754d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:30:56.161541 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.161472    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a571f371-fe5b-4fc1-8abb-483a5f5cc72c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a571f371-fe5b-4fc1-8abb-483a5f5cc72c" (UID: "a571f371-fe5b-4fc1-8abb-483a5f5cc72c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:30:56.163093 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.163062    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a571f371-fe5b-4fc1-8abb-483a5f5cc72c-kube-api-access-88s84" (OuterVolumeSpecName: "kube-api-access-88s84") pod "a571f371-fe5b-4fc1-8abb-483a5f5cc72c" (UID: "a571f371-fe5b-4fc1-8abb-483a5f5cc72c"). InnerVolumeSpecName "kube-api-access-88s84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:30:56.163093 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.163073    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0273a177-f332-47df-b8f3-8384fab5754d-kube-api-access-nq77g" (OuterVolumeSpecName: "kube-api-access-nq77g") pod "0273a177-f332-47df-b8f3-8384fab5754d" (UID: "0273a177-f332-47df-b8f3-8384fab5754d"). InnerVolumeSpecName "kube-api-access-nq77g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:30:56.262495 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.262469    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0273a177-f332-47df-b8f3-8384fab5754d-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:30:56.262495 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.262492    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88s84\" (UniqueName: \"kubernetes.io/projected/a571f371-fe5b-4fc1-8abb-483a5f5cc72c-kube-api-access-88s84\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:30:56.262619 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.262502    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a571f371-fe5b-4fc1-8abb-483a5f5cc72c-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:30:56.262619 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.262510    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq77g\" (UniqueName: \"kubernetes.io/projected/0273a177-f332-47df-b8f3-8384fab5754d-kube-api-access-nq77g\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:30:56.637430 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.637398    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c382de46-b490-4955-83c7-b4848293d9fa" path="/var/lib/kubelet/pods/c382de46-b490-4955-83c7-b4848293d9fa/volumes"
Sep 08 10:30:56.985514 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.985492    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"
Sep 08 10:30:56.985926 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.985486    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx" event={"ID":"a571f371-fe5b-4fc1-8abb-483a5f5cc72c","Type":"ContainerDied","Data":"8d584945b170520af82b1706b37cf60083e1c092d59a1091bec7b7bd69961283"}
Sep 08 10:30:56.985926 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.985645    2559 scope.go:117] "RemoveContainer" containerID="1856dd4a5f67dbdd1e72d1a4121b8793dfb0282b55e517fbcf5135d5cb3b3640"
Sep 08 10:30:56.987084 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.987002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2" event={"ID":"0273a177-f332-47df-b8f3-8384fab5754d","Type":"ContainerDied","Data":"d8e6be35c4f0a8d133f3855900479555f9b71c5c6d4af7d7cf38cdbc70547f37"}
Sep 08 10:30:56.987084 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.987030    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"
Sep 08 10:30:56.995704 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:56.995686    2559 scope.go:117] "RemoveContainer" containerID="db69e8890561cf7eda7b1c99f0815418dae8851c3de6127075ca653fa0664b7b"
Sep 08 10:30:57.006140 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:57.006120    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"]
Sep 08 10:30:57.009439 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:57.009419    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86902f-749ccf8d5-brws2"]
Sep 08 10:30:57.020011 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:57.019984    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"]
Sep 08 10:30:57.023658 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:57.023638    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86902f-579f6c9674-n4rhx"]
Sep 08 10:30:58.636747 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:58.636716    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0273a177-f332-47df-b8f3-8384fab5754d" path="/var/lib/kubelet/pods/0273a177-f332-47df-b8f3-8384fab5754d/volumes"
Sep 08 10:30:58.637126 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:30:58.637093    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a571f371-fe5b-4fc1-8abb-483a5f5cc72c" path="/var/lib/kubelet/pods/a571f371-fe5b-4fc1-8abb-483a5f5cc72c/volumes"
Sep 08 10:31:02.899600 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:02.899568    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9"]
Sep 08 10:31:02.899983 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:02.899973    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c382de46-b490-4955-83c7-b4848293d9fa" containerName="revoke-keys"
Sep 08 10:31:02.900045 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:02.899987    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c382de46-b490-4955-83c7-b4848293d9fa" containerName="revoke-keys"
Sep 08 10:31:02.900045 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:02.900005    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0273a177-f332-47df-b8f3-8384fab5754d" containerName="payload-processing"
Sep 08 10:31:02.900045 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:02.900011    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0273a177-f332-47df-b8f3-8384fab5754d" containerName="payload-processing"
Sep 08 10:31:02.900045 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:02.900020    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a571f371-fe5b-4fc1-8abb-483a5f5cc72c" containerName="payload-pre-processing"
Sep 08 10:31:02.900045 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:02.900025    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a571f371-fe5b-4fc1-8abb-483a5f5cc72c" containerName="payload-pre-processing"
Sep 08 10:31:02.900195 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:02.900092    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0273a177-f332-47df-b8f3-8384fab5754d" containerName="payload-processing"
Sep 08 10:31:02.900195 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:02.900101    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a571f371-fe5b-4fc1-8abb-483a5f5cc72c" containerName="payload-pre-processing"
Sep 08 10:31:02.900195 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:02.900110    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c382de46-b490-4955-83c7-b4848293d9fa" containerName="revoke-keys"
Sep 08 10:31:02.904558 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:02.904540    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9"
Sep 08 10:31:02.907164 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:02.907146    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:31:02.907327 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:02.907147    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:31:02.910539 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:02.910518    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9"]
Sep 08 10:31:03.022849 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:03.022818    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bed2020e-024c-41c3-82e5-44d0bfa9a10b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9\" (UID: \"bed2020e-024c-41c3-82e5-44d0bfa9a10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9"
Sep 08 10:31:03.123488 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:03.123460    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bed2020e-024c-41c3-82e5-44d0bfa9a10b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9\" (UID: \"bed2020e-024c-41c3-82e5-44d0bfa9a10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9"
Sep 08 10:31:03.124026 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:03.124006    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bed2020e-024c-41c3-82e5-44d0bfa9a10b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9\" (UID: \"bed2020e-024c-41c3-82e5-44d0bfa9a10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9"
Sep 08 10:31:03.216625 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:03.216597    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9"
Sep 08 10:31:03.488766 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:03.488733    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9"]
Sep 08 10:31:03.489957 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:31:03.489931    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed2020e_024c_41c3_82e5_44d0bfa9a10b.slice/crio-3fe57329e4ee4a2821ec02732048117bfbeddc8837354fec2041d299b83b28bc WatchSource:0}: Error finding container 3fe57329e4ee4a2821ec02732048117bfbeddc8837354fec2041d299b83b28bc: Status 404 returned error can't find the container with id 3fe57329e4ee4a2821ec02732048117bfbeddc8837354fec2041d299b83b28bc
Sep 08 10:31:04.031166 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:04.031078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9" event={"ID":"bed2020e-024c-41c3-82e5-44d0bfa9a10b","Type":"ContainerStarted","Data":"064081f4993c9e12649d290cdbdbd6e7b22431c79060fafb9b085a76e9750a41"}
Sep 08 10:31:04.031166 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:04.031134    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9" event={"ID":"bed2020e-024c-41c3-82e5-44d0bfa9a10b","Type":"ContainerStarted","Data":"3fe57329e4ee4a2821ec02732048117bfbeddc8837354fec2041d299b83b28bc"}
Sep 08 10:31:04.049856 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:04.049808    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9" podStartSLOduration=2.049793765 podStartE2EDuration="2.049793765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:31:04.048033409 +0000 UTC m=+1783.889257817" watchObservedRunningTime="2026-09-08 10:31:04.049793765 +0000 UTC m=+1783.891018148"
Sep 08 10:31:05.040223 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:05.040187    2559 generic.go:356] "Generic (PLEG): container finished" podID="bed2020e-024c-41c3-82e5-44d0bfa9a10b" containerID="064081f4993c9e12649d290cdbdbd6e7b22431c79060fafb9b085a76e9750a41" exitCode=0
Sep 08 10:31:05.040616 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:05.040273    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9" event={"ID":"bed2020e-024c-41c3-82e5-44d0bfa9a10b","Type":"ContainerDied","Data":"064081f4993c9e12649d290cdbdbd6e7b22431c79060fafb9b085a76e9750a41"}
Sep 08 10:31:06.175977 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:06.175957    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9"
Sep 08 10:31:06.346891 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:06.346803    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bed2020e-024c-41c3-82e5-44d0bfa9a10b-maas-api-service-ca\") pod \"bed2020e-024c-41c3-82e5-44d0bfa9a10b\" (UID: \"bed2020e-024c-41c3-82e5-44d0bfa9a10b\") "
Sep 08 10:31:06.347271 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:06.347222    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed2020e-024c-41c3-82e5-44d0bfa9a10b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bed2020e-024c-41c3-82e5-44d0bfa9a10b" (UID: "bed2020e-024c-41c3-82e5-44d0bfa9a10b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:31:06.448510 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:06.448478    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bed2020e-024c-41c3-82e5-44d0bfa9a10b-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:31:07.052366 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:07.052337    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9"
Sep 08 10:31:07.052530 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:07.052331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9" event={"ID":"bed2020e-024c-41c3-82e5-44d0bfa9a10b","Type":"ContainerDied","Data":"3fe57329e4ee4a2821ec02732048117bfbeddc8837354fec2041d299b83b28bc"}
Sep 08 10:31:07.052530 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:07.052462    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fe57329e4ee4a2821ec02732048117bfbeddc8837354fec2041d299b83b28bc"
Sep 08 10:31:13.441875 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:13.441778    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-71a7a1-29814390-w2p88"]
Sep 08 10:31:13.442877 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:13.442848    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-71a7a1-29814390-w2p88"]
Sep 08 10:31:13.466406 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:13.466377    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"]
Sep 08 10:31:13.466660 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:13.466610    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g" podUID="3eb57f33-d29d-47c8-92cb-a293d73067fe" containerName="payload-pre-processing" containerID="cri-o://f4f0085ee5c1dbb638b176f336562bdfc122b35938817afb17f9fea9d0f82b20" gracePeriod=30
Sep 08 10:31:13.632661 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:13.632621    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9"]
Sep 08 10:31:13.635631 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:13.635604    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-71a7a1-b6pn9"]
Sep 08 10:31:13.731095 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:13.731073    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"
Sep 08 10:31:13.912180 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:13.912152    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jms6m\" (UniqueName: \"kubernetes.io/projected/3eb57f33-d29d-47c8-92cb-a293d73067fe-kube-api-access-jms6m\") pod \"3eb57f33-d29d-47c8-92cb-a293d73067fe\" (UID: \"3eb57f33-d29d-47c8-92cb-a293d73067fe\") "
Sep 08 10:31:13.912346 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:13.912216    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb57f33-d29d-47c8-92cb-a293d73067fe-plugins-config-volume\") pod \"3eb57f33-d29d-47c8-92cb-a293d73067fe\" (UID: \"3eb57f33-d29d-47c8-92cb-a293d73067fe\") "
Sep 08 10:31:13.912595 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:13.912572    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb57f33-d29d-47c8-92cb-a293d73067fe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3eb57f33-d29d-47c8-92cb-a293d73067fe" (UID: "3eb57f33-d29d-47c8-92cb-a293d73067fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:31:13.914257 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:13.914226    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb57f33-d29d-47c8-92cb-a293d73067fe-kube-api-access-jms6m" (OuterVolumeSpecName: "kube-api-access-jms6m") pod "3eb57f33-d29d-47c8-92cb-a293d73067fe" (UID: "3eb57f33-d29d-47c8-92cb-a293d73067fe"). InnerVolumeSpecName "kube-api-access-jms6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:31:14.013944 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:14.013879    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jms6m\" (UniqueName: \"kubernetes.io/projected/3eb57f33-d29d-47c8-92cb-a293d73067fe-kube-api-access-jms6m\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:31:14.013944 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:14.013903    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb57f33-d29d-47c8-92cb-a293d73067fe-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:31:14.092731 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:14.092705    2559 generic.go:356] "Generic (PLEG): container finished" podID="3eb57f33-d29d-47c8-92cb-a293d73067fe" containerID="f4f0085ee5c1dbb638b176f336562bdfc122b35938817afb17f9fea9d0f82b20" exitCode=0
Sep 08 10:31:14.092855 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:14.092747    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"
Sep 08 10:31:14.092855 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:14.092791    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g" event={"ID":"3eb57f33-d29d-47c8-92cb-a293d73067fe","Type":"ContainerDied","Data":"f4f0085ee5c1dbb638b176f336562bdfc122b35938817afb17f9fea9d0f82b20"}
Sep 08 10:31:14.092855 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:14.092843    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g" event={"ID":"3eb57f33-d29d-47c8-92cb-a293d73067fe","Type":"ContainerDied","Data":"02076e6e5dcd077eee2c8151e9f3f55a96d89ae242b28d39bb357f784e9048c9"}
Sep 08 10:31:14.092982 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:14.092872    2559 scope.go:117] "RemoveContainer" containerID="f4f0085ee5c1dbb638b176f336562bdfc122b35938817afb17f9fea9d0f82b20"
Sep 08 10:31:14.102648 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:14.102628    2559 scope.go:117] "RemoveContainer" containerID="f4f0085ee5c1dbb638b176f336562bdfc122b35938817afb17f9fea9d0f82b20"
Sep 08 10:31:14.102949 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:31:14.102926    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f0085ee5c1dbb638b176f336562bdfc122b35938817afb17f9fea9d0f82b20\": container with ID starting with f4f0085ee5c1dbb638b176f336562bdfc122b35938817afb17f9fea9d0f82b20 not found: ID does not exist" containerID="f4f0085ee5c1dbb638b176f336562bdfc122b35938817afb17f9fea9d0f82b20"
Sep 08 10:31:14.103044 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:14.102955    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f0085ee5c1dbb638b176f336562bdfc122b35938817afb17f9fea9d0f82b20"} err="failed to get container status \"f4f0085ee5c1dbb638b176f336562bdfc122b35938817afb17f9fea9d0f82b20\": rpc error: code = NotFound desc = could not find container \"f4f0085ee5c1dbb638b176f336562bdfc122b35938817afb17f9fea9d0f82b20\": container with ID starting with f4f0085ee5c1dbb638b176f336562bdfc122b35938817afb17f9fea9d0f82b20 not found: ID does not exist"
Sep 08 10:31:14.147686 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:14.147664    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"]
Sep 08 10:31:14.158222 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:14.158197    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-71a7a1-f446cbc8b-b9j4g"]
Sep 08 10:31:14.638443 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:14.638411    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eb57f33-d29d-47c8-92cb-a293d73067fe" path="/var/lib/kubelet/pods/3eb57f33-d29d-47c8-92cb-a293d73067fe/volumes"
Sep 08 10:31:14.638811 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:14.638774    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bed2020e-024c-41c3-82e5-44d0bfa9a10b" path="/var/lib/kubelet/pods/bed2020e-024c-41c3-82e5-44d0bfa9a10b/volumes"
Sep 08 10:31:14.639049 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:14.639035    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4f2de54-024a-4a27-8b4c-3fe4108e781c" path="/var/lib/kubelet/pods/e4f2de54-024a-4a27-8b4c-3fe4108e781c/volumes"
Sep 08 10:31:20.505564 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.505438    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5"]
Sep 08 10:31:20.506059 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.506037    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb57f33-d29d-47c8-92cb-a293d73067fe" containerName="payload-pre-processing"
Sep 08 10:31:20.506141 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.506061    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb57f33-d29d-47c8-92cb-a293d73067fe" containerName="payload-pre-processing"
Sep 08 10:31:20.506141 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.506088    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed2020e-024c-41c3-82e5-44d0bfa9a10b" containerName="revoke-keys"
Sep 08 10:31:20.506141 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.506098    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed2020e-024c-41c3-82e5-44d0bfa9a10b" containerName="revoke-keys"
Sep 08 10:31:20.506319 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.506194    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bed2020e-024c-41c3-82e5-44d0bfa9a10b" containerName="revoke-keys"
Sep 08 10:31:20.506319 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.506208    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb57f33-d29d-47c8-92cb-a293d73067fe" containerName="payload-pre-processing"
Sep 08 10:31:20.509489 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.509466    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5"
Sep 08 10:31:20.511931 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.511908    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:31:20.512027 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.511915    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:31:20.516799 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.516775    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5"]
Sep 08 10:31:20.569152 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.569127    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d909e715-4d91-403e-b748-2cf3ce3764a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5\" (UID: \"d909e715-4d91-403e-b748-2cf3ce3764a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5"
Sep 08 10:31:20.669909 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.669885    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d909e715-4d91-403e-b748-2cf3ce3764a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5\" (UID: \"d909e715-4d91-403e-b748-2cf3ce3764a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5"
Sep 08 10:31:20.675751 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.675734    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 10:31:20.681443 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.681424    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d909e715-4d91-403e-b748-2cf3ce3764a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5\" (UID: \"d909e715-4d91-403e-b748-2cf3ce3764a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5"
Sep 08 10:31:20.760163 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.760108    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:31:20.763827 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.763807    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:31:20.823543 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.823522    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q7pww\""
Sep 08 10:31:20.831465 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:20.831447    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5"
Sep 08 10:31:21.045987 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:21.045896    2559 scope.go:117] "RemoveContainer" containerID="f8b06be308b4993ec2f8512cc6b186ff73f043993731dddd73a09d7053377ba2"
Sep 08 10:31:21.104492 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:21.104471    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5"]
Sep 08 10:31:21.105396 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:31:21.105372    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd909e715_4d91_403e_b748_2cf3ce3764a3.slice/crio-5845af0c0f9006e7598822f7f2fe105cb3df41431f0e6ca5f5dc72c3e8aca3a5 WatchSource:0}: Error finding container 5845af0c0f9006e7598822f7f2fe105cb3df41431f0e6ca5f5dc72c3e8aca3a5: Status 404 returned error can't find the container with id 5845af0c0f9006e7598822f7f2fe105cb3df41431f0e6ca5f5dc72c3e8aca3a5
Sep 08 10:31:21.136009 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:21.135977    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5" event={"ID":"d909e715-4d91-403e-b748-2cf3ce3764a3","Type":"ContainerStarted","Data":"5845af0c0f9006e7598822f7f2fe105cb3df41431f0e6ca5f5dc72c3e8aca3a5"}
Sep 08 10:31:22.143643 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:22.143545    2559 generic.go:356] "Generic (PLEG): container finished" podID="d909e715-4d91-403e-b748-2cf3ce3764a3" containerID="e9fe97e5af8e2589a3e99409ab9e112b16cb4afecb86dbdf7626425da6bf030e" exitCode=0
Sep 08 10:31:22.143643 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:22.143608    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5" event={"ID":"d909e715-4d91-403e-b748-2cf3ce3764a3","Type":"ContainerDied","Data":"e9fe97e5af8e2589a3e99409ab9e112b16cb4afecb86dbdf7626425da6bf030e"}
Sep 08 10:31:23.279151 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:23.279122    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5"
Sep 08 10:31:23.293115 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:23.293089    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d909e715-4d91-403e-b748-2cf3ce3764a3-maas-api-service-ca\") pod \"d909e715-4d91-403e-b748-2cf3ce3764a3\" (UID: \"d909e715-4d91-403e-b748-2cf3ce3764a3\") "
Sep 08 10:31:23.293446 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:23.293423    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d909e715-4d91-403e-b748-2cf3ce3764a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d909e715-4d91-403e-b748-2cf3ce3764a3" (UID: "d909e715-4d91-403e-b748-2cf3ce3764a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:31:23.394413 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:23.394386    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d909e715-4d91-403e-b748-2cf3ce3764a3-maas-api-service-ca\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:31:24.155235 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:24.155205    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5"
Sep 08 10:31:24.155435 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:24.155264    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5" event={"ID":"d909e715-4d91-403e-b748-2cf3ce3764a3","Type":"ContainerDied","Data":"5845af0c0f9006e7598822f7f2fe105cb3df41431f0e6ca5f5dc72c3e8aca3a5"}
Sep 08 10:31:24.155435 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:24.155312    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5845af0c0f9006e7598822f7f2fe105cb3df41431f0e6ca5f5dc72c3e8aca3a5"
Sep 08 10:31:31.262624 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.262589    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"]
Sep 08 10:31:31.263146 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.262789    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd" podUID="fc275459-5945-43bf-9575-8a20739f5ce6" containerName="payload-processing" containerID="cri-o://747cd23ce6a1dbff0e3b5ce96f019b393765d5e2aa74df97db8d2f60d5f047a9" gracePeriod=30
Sep 08 10:31:31.272520 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.272301    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"]
Sep 08 10:31:31.272656 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.272533    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs" podUID="98c49713-ae56-4458-97f9-581561fe431e" containerName="payload-pre-processing" containerID="cri-o://d7758752e522e0d4e7ea9b63fda6717eeea7bd068a6f040fafd73b4eab67db45" gracePeriod=30
Sep 08 10:31:31.538527 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.538498    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"
Sep 08 10:31:31.551570 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.551544    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"
Sep 08 10:31:31.553087 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.553062    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5"]
Sep 08 10:31:31.562502 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.562479    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6e53c-555w5"]
Sep 08 10:31:31.562583 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.562562    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bwm4\" (UniqueName: \"kubernetes.io/projected/fc275459-5945-43bf-9575-8a20739f5ce6-kube-api-access-8bwm4\") pod \"fc275459-5945-43bf-9575-8a20739f5ce6\" (UID: \"fc275459-5945-43bf-9575-8a20739f5ce6\") "
Sep 08 10:31:31.562708 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.562693    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc275459-5945-43bf-9575-8a20739f5ce6-plugins-config-volume\") pod \"fc275459-5945-43bf-9575-8a20739f5ce6\" (UID: \"fc275459-5945-43bf-9575-8a20739f5ce6\") "
Sep 08 10:31:31.563001 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.562974    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc275459-5945-43bf-9575-8a20739f5ce6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc275459-5945-43bf-9575-8a20739f5ce6" (UID: "fc275459-5945-43bf-9575-8a20739f5ce6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:31:31.564610 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.564590    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc275459-5945-43bf-9575-8a20739f5ce6-kube-api-access-8bwm4" (OuterVolumeSpecName: "kube-api-access-8bwm4") pod "fc275459-5945-43bf-9575-8a20739f5ce6" (UID: "fc275459-5945-43bf-9575-8a20739f5ce6"). InnerVolumeSpecName "kube-api-access-8bwm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:31:31.664129 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.664097    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rmvg\" (UniqueName: \"kubernetes.io/projected/98c49713-ae56-4458-97f9-581561fe431e-kube-api-access-7rmvg\") pod \"98c49713-ae56-4458-97f9-581561fe431e\" (UID: \"98c49713-ae56-4458-97f9-581561fe431e\") "
Sep 08 10:31:31.664323 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.664169    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c49713-ae56-4458-97f9-581561fe431e-plugins-config-volume\") pod \"98c49713-ae56-4458-97f9-581561fe431e\" (UID: \"98c49713-ae56-4458-97f9-581561fe431e\") "
Sep 08 10:31:31.664441 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.664424    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc275459-5945-43bf-9575-8a20739f5ce6-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:31:31.664505 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.664446    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bwm4\" (UniqueName: \"kubernetes.io/projected/fc275459-5945-43bf-9575-8a20739f5ce6-kube-api-access-8bwm4\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:31:31.664586 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.664564    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c49713-ae56-4458-97f9-581561fe431e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98c49713-ae56-4458-97f9-581561fe431e" (UID: "98c49713-ae56-4458-97f9-581561fe431e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 10:31:31.666171 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.666153    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c49713-ae56-4458-97f9-581561fe431e-kube-api-access-7rmvg" (OuterVolumeSpecName: "kube-api-access-7rmvg") pod "98c49713-ae56-4458-97f9-581561fe431e" (UID: "98c49713-ae56-4458-97f9-581561fe431e"). InnerVolumeSpecName "kube-api-access-7rmvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:31:31.765441 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.765415    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rmvg\" (UniqueName: \"kubernetes.io/projected/98c49713-ae56-4458-97f9-581561fe431e-kube-api-access-7rmvg\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:31:31.765441 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:31.765437    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c49713-ae56-4458-97f9-581561fe431e-plugins-config-volume\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:31:32.201529 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.201496    2559 generic.go:356] "Generic (PLEG): container finished" podID="98c49713-ae56-4458-97f9-581561fe431e" containerID="d7758752e522e0d4e7ea9b63fda6717eeea7bd068a6f040fafd73b4eab67db45" exitCode=0
Sep 08 10:31:32.201712 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.201537    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"
Sep 08 10:31:32.201712 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.201582    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs" event={"ID":"98c49713-ae56-4458-97f9-581561fe431e","Type":"ContainerDied","Data":"d7758752e522e0d4e7ea9b63fda6717eeea7bd068a6f040fafd73b4eab67db45"}
Sep 08 10:31:32.201712 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.201638    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs" event={"ID":"98c49713-ae56-4458-97f9-581561fe431e","Type":"ContainerDied","Data":"76f2060080de53eee36f76139ae7a4e235707fc463c42e76f3ec433803b2bf4c"}
Sep 08 10:31:32.201712 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.201666    2559 scope.go:117] "RemoveContainer" containerID="d7758752e522e0d4e7ea9b63fda6717eeea7bd068a6f040fafd73b4eab67db45"
Sep 08 10:31:32.203109 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.203083    2559 generic.go:356] "Generic (PLEG): container finished" podID="fc275459-5945-43bf-9575-8a20739f5ce6" containerID="747cd23ce6a1dbff0e3b5ce96f019b393765d5e2aa74df97db8d2f60d5f047a9" exitCode=0
Sep 08 10:31:32.203210 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.203126    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"
Sep 08 10:31:32.203210 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.203162    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd" event={"ID":"fc275459-5945-43bf-9575-8a20739f5ce6","Type":"ContainerDied","Data":"747cd23ce6a1dbff0e3b5ce96f019b393765d5e2aa74df97db8d2f60d5f047a9"}
Sep 08 10:31:32.203210 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.203205    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd" event={"ID":"fc275459-5945-43bf-9575-8a20739f5ce6","Type":"ContainerDied","Data":"bc2d24cd9350f1208618cca15a4d00c2f4718dba33f113cc420ef76cfe429a3e"}
Sep 08 10:31:32.214790 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.214758    2559 scope.go:117] "RemoveContainer" containerID="d7758752e522e0d4e7ea9b63fda6717eeea7bd068a6f040fafd73b4eab67db45"
Sep 08 10:31:32.215077 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:31:32.215049    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7758752e522e0d4e7ea9b63fda6717eeea7bd068a6f040fafd73b4eab67db45\": container with ID starting with d7758752e522e0d4e7ea9b63fda6717eeea7bd068a6f040fafd73b4eab67db45 not found: ID does not exist" containerID="d7758752e522e0d4e7ea9b63fda6717eeea7bd068a6f040fafd73b4eab67db45"
Sep 08 10:31:32.215178 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.215086    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7758752e522e0d4e7ea9b63fda6717eeea7bd068a6f040fafd73b4eab67db45"} err="failed to get container status \"d7758752e522e0d4e7ea9b63fda6717eeea7bd068a6f040fafd73b4eab67db45\": rpc error: code = NotFound desc = could not find container \"d7758752e522e0d4e7ea9b63fda6717eeea7bd068a6f040fafd73b4eab67db45\": container with ID starting with d7758752e522e0d4e7ea9b63fda6717eeea7bd068a6f040fafd73b4eab67db45 not found: ID does not exist"
Sep 08 10:31:32.215178 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.215110    2559 scope.go:117] "RemoveContainer" containerID="747cd23ce6a1dbff0e3b5ce96f019b393765d5e2aa74df97db8d2f60d5f047a9"
Sep 08 10:31:32.223079 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.223061    2559 scope.go:117] "RemoveContainer" containerID="747cd23ce6a1dbff0e3b5ce96f019b393765d5e2aa74df97db8d2f60d5f047a9"
Sep 08 10:31:32.223328 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:31:32.223311    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747cd23ce6a1dbff0e3b5ce96f019b393765d5e2aa74df97db8d2f60d5f047a9\": container with ID starting with 747cd23ce6a1dbff0e3b5ce96f019b393765d5e2aa74df97db8d2f60d5f047a9 not found: ID does not exist" containerID="747cd23ce6a1dbff0e3b5ce96f019b393765d5e2aa74df97db8d2f60d5f047a9"
Sep 08 10:31:32.223379 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.223335    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747cd23ce6a1dbff0e3b5ce96f019b393765d5e2aa74df97db8d2f60d5f047a9"} err="failed to get container status \"747cd23ce6a1dbff0e3b5ce96f019b393765d5e2aa74df97db8d2f60d5f047a9\": rpc error: code = NotFound desc = could not find container \"747cd23ce6a1dbff0e3b5ce96f019b393765d5e2aa74df97db8d2f60d5f047a9\": container with ID starting with 747cd23ce6a1dbff0e3b5ce96f019b393765d5e2aa74df97db8d2f60d5f047a9 not found: ID does not exist"
Sep 08 10:31:32.227098 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.227076    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"]
Sep 08 10:31:32.231747 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.231727    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6e53c-874676986-h6hcs"]
Sep 08 10:31:32.245683 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.245659    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"]
Sep 08 10:31:32.249970 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.249953    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d6e53c-6d865cd45-rg6rd"]
Sep 08 10:31:32.643299 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.643210    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98c49713-ae56-4458-97f9-581561fe431e" path="/var/lib/kubelet/pods/98c49713-ae56-4458-97f9-581561fe431e/volumes"
Sep 08 10:31:32.643687 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.643667    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d909e715-4d91-403e-b748-2cf3ce3764a3" path="/var/lib/kubelet/pods/d909e715-4d91-403e-b748-2cf3ce3764a3/volumes"
Sep 08 10:31:32.644117 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:31:32.644102    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc275459-5945-43bf-9575-8a20739f5ce6" path="/var/lib/kubelet/pods/fc275459-5945-43bf-9575-8a20739f5ce6/volumes"
Sep 08 10:32:21.081578 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:32:21.081545    2559 scope.go:117] "RemoveContainer" containerID="b4ae0f989ade80f7c01a9a32125f6a1ff341b87200d62af31750c238e1bcdec9"
Sep 08 10:32:21.089992 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:32:21.089970    2559 scope.go:117] "RemoveContainer" containerID="f8311f1a729664faa081dccab4bba56c335fd2899227665bf7aac691e213aac3"
Sep 08 10:32:21.097985 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:32:21.097953    2559 scope.go:117] "RemoveContainer" containerID="bf7f2acf6448fe3c3793599713c43e94b556fb41e1e59426b44b4443c817ffa0"
Sep 08 10:36:15.151084 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.151044    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"]
Sep 08 10:36:15.151732 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.151711    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc275459-5945-43bf-9575-8a20739f5ce6" containerName="payload-processing"
Sep 08 10:36:15.151780 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.151738    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc275459-5945-43bf-9575-8a20739f5ce6" containerName="payload-processing"
Sep 08 10:36:15.151780 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.151753    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d909e715-4d91-403e-b748-2cf3ce3764a3" containerName="revoke-keys"
Sep 08 10:36:15.151780 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.151762    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d909e715-4d91-403e-b748-2cf3ce3764a3" containerName="revoke-keys"
Sep 08 10:36:15.151877 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.151812    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98c49713-ae56-4458-97f9-581561fe431e" containerName="payload-pre-processing"
Sep 08 10:36:15.151877 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.151821    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="98c49713-ae56-4458-97f9-581561fe431e" containerName="payload-pre-processing"
Sep 08 10:36:15.151943 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.151929    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d909e715-4d91-403e-b748-2cf3ce3764a3" containerName="revoke-keys"
Sep 08 10:36:15.151943 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.151939    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc275459-5945-43bf-9575-8a20739f5ce6" containerName="payload-processing"
Sep 08 10:36:15.152008 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.151949    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="98c49713-ae56-4458-97f9-581561fe431e" containerName="payload-pre-processing"
Sep 08 10:36:15.155291 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.155269    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.157757 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.157732    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-fce34d90-kserve-self-signed-certs\""
Sep 08 10:36:15.164294 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.164270    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"]
Sep 08 10:36:15.177755 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.177725    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-tls-certs\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.177886 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.177789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-tmp-dir\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.177886 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.177819    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-home\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.177886 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.177840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbzz7\" (UniqueName: \"kubernetes.io/projected/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-kube-api-access-dbzz7\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.178078 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.177932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-model-cache\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.178078 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.177972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-dshm\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.278363 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.278335    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-dshm\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.278517 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.278380    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-tls-certs\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.278517 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.278414    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-tmp-dir\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.278517 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.278436    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-home\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.278517 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.278456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbzz7\" (UniqueName: \"kubernetes.io/projected/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-kube-api-access-dbzz7\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.278517 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.278489    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-model-cache\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.278854 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.278820    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-tmp-dir\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.278963 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.278832    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-home\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.278963 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.278884    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-model-cache\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.280443 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.280426    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-dshm\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.280853 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.280834    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-tls-certs\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.286498 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.286480    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbzz7\" (UniqueName: \"kubernetes.io/projected/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-kube-api-access-dbzz7\") pod \"e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") " pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.466461 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.466436    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:15.735128 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:15.735098    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"]
Sep 08 10:36:15.736582 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:36:15.736539    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42c2c036_ea4a_4fab_a675_9fbb5c5f99d1.slice/crio-fbd97416a834139919e775bf877d3c677fa6e1667e6751a61d529fcdb11d2ca3 WatchSource:0}: Error finding container fbd97416a834139919e775bf877d3c677fa6e1667e6751a61d529fcdb11d2ca3: Status 404 returned error can't find the container with id fbd97416a834139919e775bf877d3c677fa6e1667e6751a61d529fcdb11d2ca3
Sep 08 10:36:16.681807 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:16.681767    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b" event={"ID":"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1","Type":"ContainerStarted","Data":"1a56148c7201f99f1b73e58398eca153604eae07d1c5bc72906b75c948499551"}
Sep 08 10:36:16.681807 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:16.681805    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b" event={"ID":"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1","Type":"ContainerStarted","Data":"fbd97416a834139919e775bf877d3c677fa6e1667e6751a61d529fcdb11d2ca3"}
Sep 08 10:36:16.700500 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:16.700455    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b" podStartSLOduration=1.700442848 podStartE2EDuration="1.700442848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 10:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:36:16.698874417 +0000 UTC m=+2096.540098819" watchObservedRunningTime="2026-09-08 10:36:16.700442848 +0000 UTC m=+2096.541667232"
Sep 08 10:36:20.792942 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:20.792916    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:36:20.799183 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:20.799161    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:36:21.153853 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:21.153776    2559 scope.go:117] "RemoveContainer" containerID="9154b460b9461a9fa15b027e25b47a650504a3f1cf63da399ced6cd0675170fc"
Sep 08 10:36:21.162989 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:21.162972    2559 scope.go:117] "RemoveContainer" containerID="9c10518eb93d85cf69aef0813322916f8b4fd2ee7274cb058cb3133dd7efec2c"
Sep 08 10:36:21.170588 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:21.170570    2559 scope.go:117] "RemoveContainer" containerID="9a6910dd21cf2ba243551afa601c364fd46e847a9bbf283bdc10fe224747a07a"
Sep 08 10:36:25.466935 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:25.466901    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:25.467358 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:25.466968    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:25.479102 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:25.479080    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:36:25.739848 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:36:25.739775    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:37:00.669221 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:00.669185    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"]
Sep 08 10:37:15.688855 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:15.688767    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b" podUID="42c2c036-ea4a-4fab-a675-9fbb5c5f99d1" containerName="main" containerID="cri-o://1a56148c7201f99f1b73e58398eca153604eae07d1c5bc72906b75c948499551" gracePeriod=45
Sep 08 10:37:15.729551 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:15.729515    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b" podUID="42c2c036-ea4a-4fab-a675-9fbb5c5f99d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": dial tcp 10.132.0.95:8000: connect: connection refused"
Sep 08 10:37:15.938369 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:15.938337    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:37:15.985475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:15.985387    2559 generic.go:356] "Generic (PLEG): container finished" podID="42c2c036-ea4a-4fab-a675-9fbb5c5f99d1" containerID="1a56148c7201f99f1b73e58398eca153604eae07d1c5bc72906b75c948499551" exitCode=0
Sep 08 10:37:15.985475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:15.985433    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b" event={"ID":"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1","Type":"ContainerDied","Data":"1a56148c7201f99f1b73e58398eca153604eae07d1c5bc72906b75c948499551"}
Sep 08 10:37:15.985475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:15.985446    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"
Sep 08 10:37:15.985475 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:15.985473    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b" event={"ID":"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1","Type":"ContainerDied","Data":"fbd97416a834139919e775bf877d3c677fa6e1667e6751a61d529fcdb11d2ca3"}
Sep 08 10:37:15.985745 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:15.985495    2559 scope.go:117] "RemoveContainer" containerID="1a56148c7201f99f1b73e58398eca153604eae07d1c5bc72906b75c948499551"
Sep 08 10:37:15.994213 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:15.994195    2559 scope.go:117] "RemoveContainer" containerID="1a56148c7201f99f1b73e58398eca153604eae07d1c5bc72906b75c948499551"
Sep 08 10:37:15.994476 ip-10-0-128-212 kubenswrapper[2559]: E0908 10:37:15.994450    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a56148c7201f99f1b73e58398eca153604eae07d1c5bc72906b75c948499551\": container with ID starting with 1a56148c7201f99f1b73e58398eca153604eae07d1c5bc72906b75c948499551 not found: ID does not exist" containerID="1a56148c7201f99f1b73e58398eca153604eae07d1c5bc72906b75c948499551"
Sep 08 10:37:15.994526 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:15.994479    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a56148c7201f99f1b73e58398eca153604eae07d1c5bc72906b75c948499551"} err="failed to get container status \"1a56148c7201f99f1b73e58398eca153604eae07d1c5bc72906b75c948499551\": rpc error: code = NotFound desc = could not find container \"1a56148c7201f99f1b73e58398eca153604eae07d1c5bc72906b75c948499551\": container with ID starting with 1a56148c7201f99f1b73e58398eca153604eae07d1c5bc72906b75c948499551 not found: ID does not exist"
Sep 08 10:37:16.073586 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.073563    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-home\") pod \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") "
Sep 08 10:37:16.073692 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.073605    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-dshm\") pod \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") "
Sep 08 10:37:16.073692 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.073656    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-tls-certs\") pod \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") "
Sep 08 10:37:16.073692 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.073672    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-tmp-dir\") pod \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") "
Sep 08 10:37:16.073866 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.073697    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbzz7\" (UniqueName: \"kubernetes.io/projected/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-kube-api-access-dbzz7\") pod \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") "
Sep 08 10:37:16.073866 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.073767    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-model-cache\") pod \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\" (UID: \"42c2c036-ea4a-4fab-a675-9fbb5c5f99d1\") "
Sep 08 10:37:16.073866 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.073791    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-home" (OuterVolumeSpecName: "home") pod "42c2c036-ea4a-4fab-a675-9fbb5c5f99d1" (UID: "42c2c036-ea4a-4fab-a675-9fbb5c5f99d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:37:16.074019 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.073946    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42c2c036-ea4a-4fab-a675-9fbb5c5f99d1" (UID: "42c2c036-ea4a-4fab-a675-9fbb5c5f99d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:37:16.074094 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.074051    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "42c2c036-ea4a-4fab-a675-9fbb5c5f99d1" (UID: "42c2c036-ea4a-4fab-a675-9fbb5c5f99d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:37:16.074094 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.074084    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-tmp-dir\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:37:16.074206 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.074097    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-home\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:37:16.075885 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.075863    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-dshm" (OuterVolumeSpecName: "dshm") pod "42c2c036-ea4a-4fab-a675-9fbb5c5f99d1" (UID: "42c2c036-ea4a-4fab-a675-9fbb5c5f99d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 10:37:16.075966 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.075923    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-kube-api-access-dbzz7" (OuterVolumeSpecName: "kube-api-access-dbzz7") pod "42c2c036-ea4a-4fab-a675-9fbb5c5f99d1" (UID: "42c2c036-ea4a-4fab-a675-9fbb5c5f99d1"). InnerVolumeSpecName "kube-api-access-dbzz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 10:37:16.076007 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.075976    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42c2c036-ea4a-4fab-a675-9fbb5c5f99d1" (UID: "42c2c036-ea4a-4fab-a675-9fbb5c5f99d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 10:37:16.175018 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.174985    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-dshm\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:37:16.175018 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.175014    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-tls-certs\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:37:16.175018 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.175025    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbzz7\" (UniqueName: \"kubernetes.io/projected/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-kube-api-access-dbzz7\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:37:16.175188 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.175034    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1-model-cache\") on node \"ip-10-0-128-212.ec2.internal\" DevicePath \"\""
Sep 08 10:37:16.307256 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.307216    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"]
Sep 08 10:37:16.310897 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.310875    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-fce34d90-kserve-846599c6d7-vpj5b"]
Sep 08 10:37:16.636371 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:16.636294    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42c2c036-ea4a-4fab-a675-9fbb5c5f99d1" path="/var/lib/kubelet/pods/42c2c036-ea4a-4fab-a675-9fbb5c5f99d1/volumes"
Sep 08 10:37:21.204695 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:21.204666    2559 scope.go:117] "RemoveContainer" containerID="e9fe97e5af8e2589a3e99409ab9e112b16cb4afecb86dbdf7626425da6bf030e"
Sep 08 10:37:21.213233 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:21.213212    2559 scope.go:117] "RemoveContainer" containerID="064081f4993c9e12649d290cdbdbd6e7b22431c79060fafb9b085a76e9750a41"
Sep 08 10:37:21.221473 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:21.221454    2559 scope.go:117] "RemoveContainer" containerID="eeea4a12b198dbb86a9c8550829b855f2b820cf0f9aa847d0271c1f5a5782cc9"
Sep 08 10:37:51.071778 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:51.071747    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67b97c78bf-bpb97_abe11f50-eeac-4076-9669-7e0467148023/authorino/0.log"
Sep 08 10:37:56.983959 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:56.983932    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b6d9c7b-hvsvs_707073bb-7800-4be5-93d5-87d02b168c1e/copy-manifests/0.log"
Sep 08 10:37:56.991155 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:56.991117    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b6d9c7b-hvsvs_707073bb-7800-4be5-93d5-87d02b168c1e/manager/0.log"
Sep 08 10:37:57.833067 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:57.833035    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814375-lzhph_ad5bedf1-9180-49a3-b5ed-5471893ebc1f/cleanup/0.log"
Sep 08 10:37:57.942516 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:57.942488    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814390-9btzm_f21f91ac-d00c-496e-b9ac-eb5411386599/cleanup/0.log"
Sep 08 10:37:58.880960 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:58.880893    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67b97c78bf-bpb97_abe11f50-eeac-4076-9669-7e0467148023/authorino/0.log"
Sep 08 10:37:59.597091 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:37:59.597062    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-qztkm_57246cb3-882d-4ba1-ba62-cb04e246d315/manager/0.log"
Sep 08 10:38:00.210163 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:00.210135    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c9fc4cdc-fvm6d_09c3d1b2-6a41-44d0-8413-9f862a16e6ab/kube-auth-proxy/0.log"
Sep 08 10:38:00.703308 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:00.703275    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7ld5x_31528300-0a49-409d-b0c4-f3083656cc48/payload-processing/0.log"
Sep 08 10:38:00.818781 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:00.818752    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fbcc7c578-fzqlh_132efab2-1cc6-4fac-a9ef-61d9d8ab4c48/router/0.log"
Sep 08 10:38:01.417649 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:01.417619    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-fg58d_5387c48a-c2f8-4ce6-97f7-be088db3ad62/main/0.log"
Sep 08 10:38:01.780687 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:01.780609    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-l24xl_45073922-69bd-4ce6-b2f7-667f8e0ef467/main/0.log"
Sep 08 10:38:01.904854 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:01.904802    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-x77c9_e91e8a04-b137-4873-91f4-4494e1726297/main/0.log"
Sep 08 10:38:10.769920 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:10.769892    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9h7fv_81df3687-05a6-4b77-ac67-2f8c6626e906/global-pull-secret-syncer/0.log"
Sep 08 10:38:10.992298 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:10.992267    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-w8khh_62ec76ef-d924-4dbb-b66b-735e295ddc0c/konnectivity-agent/0.log"
Sep 08 10:38:11.025191 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:11.025127    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-212.ec2.internal_052f4ba795232d6489bc9a1bde394dc3/haproxy/0.log"
Sep 08 10:38:15.436717 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:15.436685    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67b97c78bf-bpb97_abe11f50-eeac-4076-9669-7e0467148023/authorino/0.log"
Sep 08 10:38:15.684258 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:15.684220    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-qztkm_57246cb3-882d-4ba1-ba62-cb04e246d315/manager/0.log"
Sep 08 10:38:17.489102 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:17.489070    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-wgqz8_bd18742f-fe5e-4fcb-b416-b566be8344e4/cluster-monitoring-operator/0.log"
Sep 08 10:38:17.534607 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:17.534577    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-xvrdg_ac488f0b-dab6-4b4a-8227-de0f72e47555/kube-state-metrics/0.log"
Sep 08 10:38:17.568395 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:17.568370    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-xvrdg_ac488f0b-dab6-4b4a-8227-de0f72e47555/kube-rbac-proxy-main/0.log"
Sep 08 10:38:17.590705 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:17.590682    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-xvrdg_ac488f0b-dab6-4b4a-8227-de0f72e47555/kube-rbac-proxy-self/0.log"
Sep 08 10:38:17.646820 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:17.646797    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-trzkt_0190d889-1bbf-4260-a6b7-275de53ebbd8/monitoring-plugin/0.log"
Sep 08 10:38:17.807579 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:17.807506    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5djbs_213ddfa0-a642-495a-a1f0-3038d0e2a5c4/node-exporter/0.log"
Sep 08 10:38:17.830815 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:17.830761    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5djbs_213ddfa0-a642-495a-a1f0-3038d0e2a5c4/kube-rbac-proxy/0.log"
Sep 08 10:38:17.861525 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:17.861504    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5djbs_213ddfa0-a642-495a-a1f0-3038d0e2a5c4/init-textfile/0.log"
Sep 08 10:38:18.099689 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:18.099604    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce127e3c-6c5b-413c-b506-fc36dac79ae2/prometheus/0.log"
Sep 08 10:38:18.121277 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:18.121232    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce127e3c-6c5b-413c-b506-fc36dac79ae2/config-reloader/0.log"
Sep 08 10:38:18.143499 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:18.143461    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce127e3c-6c5b-413c-b506-fc36dac79ae2/thanos-sidecar/0.log"
Sep 08 10:38:18.169398 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:18.169376    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce127e3c-6c5b-413c-b506-fc36dac79ae2/kube-rbac-proxy-web/0.log"
Sep 08 10:38:18.187376 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:18.187341    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce127e3c-6c5b-413c-b506-fc36dac79ae2/kube-rbac-proxy/0.log"
Sep 08 10:38:18.213893 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:18.213869    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce127e3c-6c5b-413c-b506-fc36dac79ae2/kube-rbac-proxy-thanos/0.log"
Sep 08 10:38:18.235995 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:18.235977    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce127e3c-6c5b-413c-b506-fc36dac79ae2/init-config-reloader/0.log"
Sep 08 10:38:18.352038 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:18.351977    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8ff4d8c67-bb5t9_eead0fa1-afe9-4343-979e-bf006a2dce3c/telemeter-client/0.log"
Sep 08 10:38:18.383103 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:18.383080    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8ff4d8c67-bb5t9_eead0fa1-afe9-4343-979e-bf006a2dce3c/reload/0.log"
Sep 08 10:38:18.405339 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:18.405318    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8ff4d8c67-bb5t9_eead0fa1-afe9-4343-979e-bf006a2dce3c/kube-rbac-proxy/0.log"
Sep 08 10:38:19.904814 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:19.904786    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-7cwbx_13d91b33-2bf0-4bf4-b866-8b6bc251c280/networking-console-plugin/0.log"
Sep 08 10:38:20.515507 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:20.515481    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/1.log"
Sep 08 10:38:20.522497 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:20.522474    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pm7xz_4e917bce-6da7-4d7a-bbb2-bd09d8086599/console-operator/2.log"
Sep 08 10:38:21.137390 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.137358    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"]
Sep 08 10:38:21.137761 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.137707    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c2c036-ea4a-4fab-a675-9fbb5c5f99d1" containerName="main"
Sep 08 10:38:21.137761 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.137718    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c2c036-ea4a-4fab-a675-9fbb5c5f99d1" containerName="main"
Sep 08 10:38:21.137839 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.137787    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c2c036-ea4a-4fab-a675-9fbb5c5f99d1" containerName="main"
Sep 08 10:38:21.141115 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.141094    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.143700 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.143678    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q6fc7\"/\"openshift-service-ca.crt\""
Sep 08 10:38:21.143811 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.143686    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q6fc7\"/\"kube-root-ca.crt\""
Sep 08 10:38:21.144597 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.144579    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q6fc7\"/\"default-dockercfg-hchl5\""
Sep 08 10:38:21.151943 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.151919    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"]
Sep 08 10:38:21.213935 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.213909    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc2sr\" (UniqueName: \"kubernetes.io/projected/70ad86fa-6942-444a-a07c-a6cab30b88dc-kube-api-access-fc2sr\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.214035 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.213950    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70ad86fa-6942-444a-a07c-a6cab30b88dc-proc\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.214035 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.213988    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70ad86fa-6942-444a-a07c-a6cab30b88dc-sys\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.214035 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.214011    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70ad86fa-6942-444a-a07c-a6cab30b88dc-lib-modules\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.214144 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.214041    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70ad86fa-6942-444a-a07c-a6cab30b88dc-podres\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.314811 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.314786    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc2sr\" (UniqueName: \"kubernetes.io/projected/70ad86fa-6942-444a-a07c-a6cab30b88dc-kube-api-access-fc2sr\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.314934 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.314828    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70ad86fa-6942-444a-a07c-a6cab30b88dc-proc\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.314934 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.314855    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70ad86fa-6942-444a-a07c-a6cab30b88dc-sys\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.314934 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.314890    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70ad86fa-6942-444a-a07c-a6cab30b88dc-lib-modules\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.314934 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.314917    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70ad86fa-6942-444a-a07c-a6cab30b88dc-proc\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.315136 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.314956    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70ad86fa-6942-444a-a07c-a6cab30b88dc-podres\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.315136 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.314997    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70ad86fa-6942-444a-a07c-a6cab30b88dc-sys\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.315136 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.315048    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70ad86fa-6942-444a-a07c-a6cab30b88dc-lib-modules\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.315136 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.315071    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70ad86fa-6942-444a-a07c-a6cab30b88dc-podres\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.322865 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.322847    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc2sr\" (UniqueName: \"kubernetes.io/projected/70ad86fa-6942-444a-a07c-a6cab30b88dc-kube-api-access-fc2sr\") pod \"perf-node-gather-daemonset-nsbb9\" (UID: \"70ad86fa-6942-444a-a07c-a6cab30b88dc\") " pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.451988 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.451968    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:21.723099 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.723071    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"]
Sep 08 10:38:21.724918 ip-10-0-128-212 kubenswrapper[2559]: W0908 10:38:21.724894    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod70ad86fa_6942_444a_a07c_a6cab30b88dc.slice/crio-150183fc487a7dd9d120d2b04825eccdd49a34d590c1262a1e7e1a72fb211bc7 WatchSource:0}: Error finding container 150183fc487a7dd9d120d2b04825eccdd49a34d590c1262a1e7e1a72fb211bc7: Status 404 returned error can't find the container with id 150183fc487a7dd9d120d2b04825eccdd49a34d590c1262a1e7e1a72fb211bc7
Sep 08 10:38:21.834638 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:21.834613    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-hpbwn_15e7a59c-e003-45d4-8878-63728f83b9dc/volume-data-source-validator/0.log"
Sep 08 10:38:22.319006 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:22.318977    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9" event={"ID":"70ad86fa-6942-444a-a07c-a6cab30b88dc","Type":"ContainerStarted","Data":"9d9a676313c1bc6396e7c122b909c425ad4d338b17bec30261102035d68d6d88"}
Sep 08 10:38:22.319476 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:22.319030    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:22.319476 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:22.319053    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9" event={"ID":"70ad86fa-6942-444a-a07c-a6cab30b88dc","Type":"ContainerStarted","Data":"150183fc487a7dd9d120d2b04825eccdd49a34d590c1262a1e7e1a72fb211bc7"}
Sep 08 10:38:22.336845 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:22.336799    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9" podStartSLOduration=1.336785698 podStartE2EDuration="1.336785698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 10:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 10:38:22.334948796 +0000 UTC m=+2222.176173204" watchObservedRunningTime="2026-09-08 10:38:22.336785698 +0000 UTC m=+2222.178010078"
Sep 08 10:38:23.075215 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:23.075186    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fnmlf_cacaccdd-dd0d-4bcd-a974-a065758240f7/dns/0.log"
Sep 08 10:38:23.100979 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:23.100953    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fnmlf_cacaccdd-dd0d-4bcd-a974-a065758240f7/kube-rbac-proxy/0.log"
Sep 08 10:38:23.229981 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:23.229956    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5lzjn_ff849db0-791c-405b-bbfc-868f1b7288d2/dns-node-resolver/0.log"
Sep 08 10:38:23.879716 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:23.879689    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hp9jt_2719b26a-ad31-4018-a46c-927706ebc26e/node-ca/0.log"
Sep 08 10:38:25.198518 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:25.198491    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c9fc4cdc-fvm6d_09c3d1b2-6a41-44d0-8413-9f862a16e6ab/kube-auth-proxy/0.log"
Sep 08 10:38:25.361881 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:25.361837    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7ld5x_31528300-0a49-409d-b0c4-f3083656cc48/payload-processing/0.log"
Sep 08 10:38:25.391038 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:25.391013    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fbcc7c578-fzqlh_132efab2-1cc6-4fac-a9ef-61d9d8ab4c48/router/0.log"
Sep 08 10:38:26.228459 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:26.228428    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5h7jm_2461e11a-d91a-4e42-bb9e-8646646fdf40/serve-healthcheck-canary/0.log"
Sep 08 10:38:26.777649 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:26.777619    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-dwmlf_1ba9dc85-d817-4dde-bb41-b50a8fd3d544/insights-operator/0.log"
Sep 08 10:38:26.777922 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:26.777907    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-dwmlf_1ba9dc85-d817-4dde-bb41-b50a8fd3d544/insights-operator/1.log"
Sep 08 10:38:27.061496 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:27.061426    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wvgzg_d251d8ad-8bef-4a17-8cd3-7da32a35deb6/kube-rbac-proxy/0.log"
Sep 08 10:38:27.121128 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:27.121110    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wvgzg_d251d8ad-8bef-4a17-8cd3-7da32a35deb6/exporter/0.log"
Sep 08 10:38:27.176623 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:27.176599    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wvgzg_d251d8ad-8bef-4a17-8cd3-7da32a35deb6/extractor/0.log"
Sep 08 10:38:28.336189 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:28.336163    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q6fc7/perf-node-gather-daemonset-nsbb9"
Sep 08 10:38:29.816807 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:29.816774    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b6d9c7b-hvsvs_707073bb-7800-4be5-93d5-87d02b168c1e/manager/0.log"
Sep 08 10:38:29.838213 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:29.838188    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b6d9c7b-hvsvs_707073bb-7800-4be5-93d5-87d02b168c1e/copy-manifests/0.log"
Sep 08 10:38:31.263375 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:31.263294    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c9fc4cdc-fvm6d_09c3d1b2-6a41-44d0-8413-9f862a16e6ab/kube-auth-proxy/0.log"
Sep 08 10:38:31.344676 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:31.344646    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-86bd4f8cd9-5gbr5_3196d75c-584d-48d5-bf15-8604c9616f7a/manager/0.log"
Sep 08 10:38:31.406609 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:31.406580    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7ld5x_31528300-0a49-409d-b0c4-f3083656cc48/payload-processing/0.log"
Sep 08 10:38:31.423115 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:31.423093    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fbcc7c578-fzqlh_132efab2-1cc6-4fac-a9ef-61d9d8ab4c48/router/0.log"
Sep 08 10:38:33.369223 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:33.369197    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c9fc4cdc-fvm6d_09c3d1b2-6a41-44d0-8413-9f862a16e6ab/kube-auth-proxy/0.log"
Sep 08 10:38:33.505031 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:33.505004    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7ld5x_31528300-0a49-409d-b0c4-f3083656cc48/payload-processing/0.log"
Sep 08 10:38:33.524067 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:33.524037    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fbcc7c578-fzqlh_132efab2-1cc6-4fac-a9ef-61d9d8ab4c48/router/0.log"
Sep 08 10:38:36.653301 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:36.653267    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-9r4k7_921284cb-1a60-4623-80bf-e2c32870de7d/migrator/0.log"
Sep 08 10:38:36.675192 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:36.675171    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-9r4k7_921284cb-1a60-4623-80bf-e2c32870de7d/graceful-termination/0.log"
Sep 08 10:38:37.045571 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:37.045529    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-ls69q_5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9/kube-storage-version-migrator-operator/1.log"
Sep 08 10:38:37.047158 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:37.047131    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-ls69q_5eba85f8-a8d6-4c6b-ad15-bbf9e20e11f9/kube-storage-version-migrator-operator/0.log"
Sep 08 10:38:38.187165 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:38.187134    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8sj86_fdf41754-0a6d-4c9c-93c5-240108b5c9db/kube-multus/0.log"
Sep 08 10:38:38.212620 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:38.212597    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4jxq2_59f5177b-f1ce-4ef1-b7c2-752e58d5e27b/kube-multus-additional-cni-plugins/0.log"
Sep 08 10:38:38.234793 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:38.234771    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4jxq2_59f5177b-f1ce-4ef1-b7c2-752e58d5e27b/egress-router-binary-copy/0.log"
Sep 08 10:38:38.257849 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:38.257829    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4jxq2_59f5177b-f1ce-4ef1-b7c2-752e58d5e27b/cni-plugins/0.log"
Sep 08 10:38:38.283493 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:38.283471    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4jxq2_59f5177b-f1ce-4ef1-b7c2-752e58d5e27b/bond-cni-plugin/0.log"
Sep 08 10:38:38.307373 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:38.307345    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4jxq2_59f5177b-f1ce-4ef1-b7c2-752e58d5e27b/routeoverride-cni/0.log"
Sep 08 10:38:38.327161 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:38.327142    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4jxq2_59f5177b-f1ce-4ef1-b7c2-752e58d5e27b/whereabouts-cni-bincopy/0.log"
Sep 08 10:38:38.349753 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:38.349722    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4jxq2_59f5177b-f1ce-4ef1-b7c2-752e58d5e27b/whereabouts-cni/0.log"
Sep 08 10:38:38.784439 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:38.784409    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2pgcf_a8a40235-67e9-4b28-ad0c-97751e2afe10/network-metrics-daemon/0.log"
Sep 08 10:38:38.800744 ip-10-0-128-212 kubenswrapper[2559]: I0908 10:38:38.800691    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2pgcf_a8a40235-67e9-4b28-ad0c-97751e2afe10/kube-rbac-proxy/0.log"